Oct 30 21:49:04.679210 ip-10-0-141-147 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:49:05.055707 ip-10-0-141-147 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:05.055707 ip-10-0-141-147 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:49:05.055707 ip-10-0-141-147 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:05.055707 ip-10-0-141-147 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:05.059101 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059020    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.059101 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059100    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.059101 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059104    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059107    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059111    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059114    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059117    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059120    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059122    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059125    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059128    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059131    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059133    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059136    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059138    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059141    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059144    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059147    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059149    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059152    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059160    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059163    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.059196 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059165    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059168    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059170    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059173    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059175    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059178    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059181    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059183    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059185    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059188    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059191    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059193    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059195    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059198    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059201    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059205    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059211    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059214    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059217    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.059662 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059219    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059222    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059225    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059228    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059231    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059234    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059236    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059239    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059242    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059244    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059248    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059250    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059253    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059255    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059258    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059261    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059264    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059266    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059269    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059271    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.060119 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059274    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059277    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059280    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059284    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059287    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059290    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059292    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059295    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059298    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059300    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059303    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059305    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059307    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059310    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059313    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059315    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059317    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059320    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059322    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.060607 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059325    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059328    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059330    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059333    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059335    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059338    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059340    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059343    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059346    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059349    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059352    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059354    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059357    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059359    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059362    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059364    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059367    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059369    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059371    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.061075 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059374    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059376    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059378    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059381    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059384    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059387    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.059389    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060221    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060229    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060231    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060234    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060237    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060240    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060243    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060246    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060249    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060253    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060257    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060261    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060265    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.061519 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060270    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060276    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060280    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060285    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060288    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060291    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060294    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060297    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060299    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060302    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060305    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060307    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060311    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060315    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060318    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060320    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060323    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060325    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060328    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.062008 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060330    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060333    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060335    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060338    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060341    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060343    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060345    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060348    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060352    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060355    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060357    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060359    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060362    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060366    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060369    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060372    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060374    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060377    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060379    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.062461 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060382    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060384    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060387    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060389    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060392    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060394    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060397    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060399    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060402    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060413    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060416    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060418    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060421    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060424    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060426    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060429    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060432    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060434    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060437    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060439    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.062905 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060442    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060444    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060447    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060450    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060452    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060455    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060457    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060461    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060463    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060466    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060468    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060471    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060473    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060476    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060478    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060481    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060483    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060486    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060488    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.063390 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060491    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060493    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060495    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060498    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060500    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060502    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060505    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060507    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060510    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060512    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060514    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060517    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060519    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060522    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060524    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.060530    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061026    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061033    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061035    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.063834 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061039    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061042    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061045    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061047    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061050    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061053    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061055    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061058    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061060    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061063    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061065    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061068    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061070    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061073    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061076    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061078    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061081    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061084    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061086    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061089    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.064378 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061092    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061094    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061097    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061100    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061103    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061106    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061108    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061110    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061113    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061116    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061119    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061121    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061124    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061127    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061130    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061132    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061135    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061137    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061140    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061142    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.064869 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061145    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061147    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061149    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061152    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061154    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061156    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061159    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061161    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061164    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061166    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061168    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061171    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061173    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061175    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061178    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061182    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061185    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061187    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061190    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.065386 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061192    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061194    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061197    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061201    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061206    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061210    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061214    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061216    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061219    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061221    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061224    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061226    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061228    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061231    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061233    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061236    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061238    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061241    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061243    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.065825 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061245    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061248    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061250    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061252    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061255    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061257    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061259    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061262    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061264    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061271    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061273    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061276    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061279    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061282    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061284    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061286    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061289    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061291    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061294    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.066287 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061297    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061299    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061301    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061304    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061306    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.061308    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063002    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063011    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063019    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063024    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063028    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063031    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063038    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063043    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063046    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063049    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063052    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063055    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063058    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063061    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063064    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063067    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063070    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:49:05.066766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063073    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063077    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063080    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063083    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063086    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063089    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063093    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063096    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063100    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063103    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063106    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063109    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063112    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063115    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063118    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063123    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063126    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063129    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063132    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063135    2572 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063138    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063142    2572 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063146    2572 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063149    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:49:05.067313 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063151    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063154    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063158    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063161    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063164    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063167    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063170    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063173    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063176    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063180    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063183    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063186    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063189    2572 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063193    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063196    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063199    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063202    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063206    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063209    2572 flags.go:64] FLAG: --help="false"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063212    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063215    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063218    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063221    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:49:05.067899 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063224    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063227    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063231    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063233    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063236    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063239    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063242    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063245    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063248    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063251    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063254    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063257    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063260    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063262    2572 flags.go:64] FLAG: --lock-file=""
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063265    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063268    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063271    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063276    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063282    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063285    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063288    2572 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063291    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063294    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063297    2572 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063300    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:49:05.068462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063304    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063307    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063312    2572 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063315    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063318    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063321    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063324    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063327    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063330    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063333    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063340    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063342    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063346    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063349    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063351    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063354    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063357    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063360    2572 flags.go:64] FLAG: --port="10250"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063363    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063366    2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fe041f1d29ca0b79"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063369    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063372    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063375    2572 flags.go:64] FLAG: --register-node="true"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063378    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063382    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063386    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:49:05.069081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063389    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063391    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063395    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063398    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063401    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063404    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063407    2572 flags.go:64] FLAG: --runonce="false"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063410    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063413    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063417    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063419    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063422    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063425    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063428    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063431    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063434    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063437    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063440    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063442    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063447    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063450    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063452    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063459    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063462    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063465    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:49:05.069690 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063469    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063471    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063474    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063477    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063480    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063483    2572 flags.go:64] FLAG: --v="2"
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063489    2572 flags.go:64] FLAG: --version="false"
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063493    2572 flags.go:64] FLAG: --vmodule=""
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063497    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063501    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063637    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063641    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063644    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063647    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063649    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063652    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063654    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063657    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063660    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063663    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063665    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063668    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.070296 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063670    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063673    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063675    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063677    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063681    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063684    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063686    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063689    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063691    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063694    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063696    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063698    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063701    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063703    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063706    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063708    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063712    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063714    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063717    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.070808 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063719    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063722    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063724    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063727    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063729    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063732    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063734    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063737    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063739    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063742    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063745    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063747    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063749    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063752    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063754    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063756    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063759    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063763    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063765    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063768    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.071317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063770    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063772    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063775    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063777    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063780    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063782    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063784    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063787    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063791    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063796    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063800    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063803    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063806    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063808    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063811    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063814    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063816    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063819    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063821    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.071798 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063824    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063827    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063829    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063832    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063834    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063837    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063839    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063842    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063844    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063847    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063850    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063853    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063855    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063858    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063860    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063863    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063865    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063867    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063870    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063872    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.072265 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063875    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063877    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063880    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063882    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063885    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063887    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063889    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063892    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063894    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063898    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063901    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063904    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063906    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063909    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063925    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063928    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.072742 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.063956    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063992    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063996    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.063999    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064002    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064005    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064009    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064012    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064014    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064017    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064019    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064022    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064025    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064027    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064030    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064032    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064034    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064037    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064039    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.073230 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064042    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064044    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064047    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064049    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064052    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064054    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064058    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064061    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064064    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064066    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064069    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064071    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064074    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064077    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064080    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064082    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064085    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064087    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064090    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.073675 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064094    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064096    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064099    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064101    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064103    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064106    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064108    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064110    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064113    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064115    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064117    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064120    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064122    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064125    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064127    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064130    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064132    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064135    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064137    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064140    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.074129 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064142    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064144    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064147    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064149    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064152    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064154    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064156    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064160    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064163    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064166    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064169    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064172    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064176    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064178    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064181    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064183    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064185    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064188    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064190    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064192    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.074596 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064195    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064197    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064200    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064203    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064205    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064208    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064210    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064213    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064215    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064217    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064220    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064223    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064225    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064227    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064230    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064232    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064234    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064237    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064239    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.075081 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064242    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064244    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064247    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064250    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064252    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064256    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064258    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064260    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064263    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.064265    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.071705    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.071720    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.072379    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.072386    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072467    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072473    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072477    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072480    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072482    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.075521 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072485    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072488    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072491    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072493    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072496    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072499    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072502    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072504    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072507    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072509    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072512    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072515    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072517    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072520    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072522    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072531    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072534    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072537    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072539    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.075983 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072542    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072544    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072546    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072549    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072552    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072555    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072557    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072560    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072563    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072565    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072568    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072571    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072573    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072582    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072585    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072587    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072590    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072592    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072595    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072597    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.076434 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072600    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072602    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072605    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072608    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072610    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072613    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072615    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072618    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072620    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072628    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072631    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072633    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072636    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072638    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072641    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072643    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072647    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072650    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072652    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072655    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.076967 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072657    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072660    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072662    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072664    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072667    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072669    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072672    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072674    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072676    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072679    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072681    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072684    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072687    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072691    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072694    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072697    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072701    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072704    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072707    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.077587 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072710    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072713    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072715    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072723    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072725    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072728    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072730    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072733    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072735    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072738    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072741    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072743    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072746    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072748    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072751    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072754    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072756    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072759    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072762    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.078057 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072764    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072766    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072769    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.072771    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.072800    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073022    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073028    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073031    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073033    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073036    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073038    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073041    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073043    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073046    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073048    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073050    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.078495 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073053    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073061    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073063    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073066    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073068    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073071    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073073    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073076    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073079    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073081    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073083    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073086    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073088    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073091    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073093    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073096    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073099    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073101    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073103    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073106    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.078865 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073108    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073111    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073113    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073116    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073118    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073120    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073123    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073126    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073128    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073130    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073133    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073135    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073137    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073140    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073148    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073151    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073153    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073156    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073158    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073161    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.079360 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073164    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073166    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073168    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073172    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073176    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073179    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073181    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073184    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073187    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073189    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073191    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073194    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073196    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073199    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073201    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073203    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073206    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073208    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073210    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.079838 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073213    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073215    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073218    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073220    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073222    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073225    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073227    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073229    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073237    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073240    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073243    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073246    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073249    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073251    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073255    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073259    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073262    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073265    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073268    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.080331 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073270    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073273    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073275    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073278    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073280    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073282    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073285    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073287    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073290    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073292    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073294    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073297    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073299    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073302    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073304    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073307    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.080802 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073309    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.073334    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073387    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073392    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073396    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073398    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073407    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073410    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073413    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073416    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073418    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073421    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073423    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073426    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073428    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073431    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:05.081278 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073433    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073436    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073438    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073440    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073443    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073445    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073447    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073450    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073452    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073455    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073457    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073460    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073462    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073464    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073467    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073469    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073471    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073474    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073476    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:05.081645 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073479    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073481    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073483    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073486    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073494    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073496    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073499    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073501    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073504    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073506    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073509    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073511    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073514    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073516    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073518    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073526    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073529    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073531    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073534    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:05.082317 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073537    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073541    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073543    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073546    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073548    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073551    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073553    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073555    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073558    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073561    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073564    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073566    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073568    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073571    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073573    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073576    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073578    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073580    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073588    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073591    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:05.083022 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073593    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073596    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073598    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073601    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073603    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073606    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073608    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073611    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073613    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073615    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073619    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073623    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073626    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073629    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073631    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073634    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073636    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073639    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:05.083483 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073641    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073643    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073646    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073648    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073651    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073653    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073656    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073659    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073661    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073664    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073666    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073668    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073671    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073678    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073682    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:05.073685    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.073740    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.076996    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.078002    2572 server.go:1014] "Starting client certificate rotation"
Oct 30 21:49:05.083944 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.078105    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:05.084396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.078152    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:05.084396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.080342    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:05.086167 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.086147    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:05.098002 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.097981    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:49:05.102887 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.102838    2572 log.go:25] "Validated CRI v1 image API"
Oct 30 21:49:05.104006 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.103974    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:49:05.106591 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.106566    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8c007e03-792c-439e-9e15-c57cad5c676c:/dev/nvme0n1p4 da8bc8a4-ffc0-4f30-a246-8cd06ab03bff:/dev/nvme0n1p3]
Oct 30 21:49:05.106656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.106591    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:49:05.112215 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.112105    2572 manager.go:217] Machine: {Timestamp:2026-10-30 21:49:05.110676607 +0000 UTC m=+0.320799010 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3074982 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258c07aff4542c531a08703f6f1208 SystemUUID:ec258c07-aff4-542c-531a-08703f6f1208 BootID:df969df4-f327-4abf-aa3a-057a4d6feac8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:90:22:e9:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:90:22:e9:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:98:e6:c8:e0:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:49:05.112215 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.112214    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:49:05.112337 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.112301    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:49:05.115650 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.115627    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:49:05.115795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.115652    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-147.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:49:05.115839 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.115797    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:49:05.115839 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.115805    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:49:05.115894 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.115841    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:05.115894 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.115859    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:05.117211 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.117200    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:49:05.124164 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.124149    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:49:05.124325 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.124315    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:49:05.125158 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.125148    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:49:05.125188 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.125163    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:49:05.125188 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.125173    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:49:05.125188 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.125182    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:49:05.125272 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.125190    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:49:05.125501 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.125488    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:05.127812 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.127796    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:49:05.135926 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.135901    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:05.136844 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.136830    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:49:05.136885 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.136862    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:49:05.137040 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.137030    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:49:05.137086 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.137046    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:49:05.137086 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.137053    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:49:05.137086 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.137058    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:49:05.137086 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.137063    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:49:05.137086 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.137069    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:49:05.138175 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.138165    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:49:05.138211 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.138177    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:49:05.138211 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.138185    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:49:05.138211 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.138190    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:49:05.138211 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.138206    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:49:05.138323 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.138215    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:49:05.139203 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.139194    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:49:05.139238 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.139204    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:49:05.143173 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.143158    2572 server.go:1263] "Started kubelet"
Oct 30 21:49:05.143234 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.143207    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:49:05.143471 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.143300    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:49:05.143573 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.143561    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:49:05.143964 ip-10-0-141-147 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:49:05.145082 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.144909    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:49:05.149405 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.149380    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:49:05.149775 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.149763    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:49:05.152366 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.152348    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:05.153279 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.153266    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:49:05.153407 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.153368    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:05.154122 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154102    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:49:05.154122 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154123    2572 factory.go:55] Registering systemd factory
Oct 30 21:49:05.154231 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154133    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:49:05.154231 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154170    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:49:05.154289 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154194    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:49:05.154289 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154250    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:49:05.154373 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154358    2572 factory.go:153] Registering CRI-O factory
Oct 30 21:49:05.154437 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154378    2572 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:49:05.154437 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154395    2572 factory.go:103] Registering Raw factory
Oct 30 21:49:05.154437 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154408    2572 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:49:05.154578 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154366    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:49:05.154578 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.154481    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:49:05.155404 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.155388    2572 manager.go:319] Starting recovery of all containers
Oct 30 21:49:05.165622 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.165604    2572 manager.go:324] Recovery completed
Oct 30 21:49:05.167328 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.167305    2572 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 21:49:05.170124 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.170112    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:05.171441 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.171419    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:05.171492 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.171461    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:05.171492 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.171471    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:05.171995 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.171982    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:49:05.171995 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.171993    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:49:05.172084 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.172004    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:05.172084 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.172011    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:49:05.172662 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.172646    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:05.172734 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.172676    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:05.172734 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.172690    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:05.174121 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.174109    2572 policy_none.go:50] "Start"
Oct 30 21:49:05.174121 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.174124    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:49:05.174222 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.174136    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:49:05.176035 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.176025    2572 policy_none.go:44] "Start"
Oct 30 21:49:05.238095 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.221027    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:49:05.238095 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.228646    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:49:05.238095 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.229891    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:49:05.238095 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.229909    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:49:05.238095 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.229941    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:49:05.238095 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.229978    2572 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:49:05.241414 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.241398    2572 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:49:05.241505 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.241428    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:49:05.241505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.241439    2572 server.go:87] "Starting device plugin registration server"
Oct 30 21:49:05.241677 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.241664    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:49:05.241738 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.241678    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:49:05.241787 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.241776    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:49:05.241866 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.241856    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:49:05.241971 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.241866    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:49:05.242319 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.242293    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:49:05.242401 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.242343    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:05.289888 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.289856    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-147.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:49:05.289888 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.289871    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-147.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:49:05.290957 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.289864    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-147.us-west-1.compute.internal.18e36d05108435c0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-147.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-147.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:05.142953408 +0000 UTC m=+0.353075810,LastTimestamp:2026-10-30 21:49:05.142953408 +0000 UTC m=+0.353075810,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-147.us-west-1.compute.internal,}"
Oct 30 21:49:05.330685 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.330638    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"]
Oct 30 21:49:05.330763 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.330718    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:05.331449 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.331433    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:05.331526 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.331467    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:05.331526 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.331483    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:05.333722 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.333707    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:05.333874 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.333858    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.333943 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.333891    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:05.334439 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.334416    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:05.334439 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.334428    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:05.334560 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.334445    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:05.334560 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.334452    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:05.334560 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.334461    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:05.334560 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.334473    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:05.336603 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.336590    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.336656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.336614    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:05.337293 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.337273    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:05.337293 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.337291    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:05.337397 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.337301    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:05.342194 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.342180    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:05.342802 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.342788    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:05.342881 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.342814    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:05.342881 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.342823    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:05.342881 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.342840    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.355894 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.355873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.356015 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.355904    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.356015 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.355976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46aa86434e4345d68e5cda43f3103b52-config\") pod \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"46aa86434e4345d68e5cda43f3103b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.366816 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.366800    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:49:05.366928 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.366864    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9gm4d"
Oct 30 21:49:05.366928 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.366884    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9gm4d"
Oct 30 21:49:05.371013 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.370996    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found" node="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.373232 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.373215    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found" node="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.390640 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.390616    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:49:05.390640 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.390616    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 21:49:05.390746 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.390622    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:49:05.390746 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.390596    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-147.us-west-1.compute.internal.18e36d051236e67e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-147.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-147.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-147.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:05.171441278 +0000 UTC m=+0.381563681,LastTimestamp:2026-10-30 21:49:05.171441278 +0000 UTC m=+0.381563681,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-147.us-west-1.compute.internal,}"
Oct 30 21:49:05.390828 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.390804    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-147.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:49:05.456292 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.456270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.456373 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.456295    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.456373 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.456312    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46aa86434e4345d68e5cda43f3103b52-config\") pod \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"46aa86434e4345d68e5cda43f3103b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.456373 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.456356    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46aa86434e4345d68e5cda43f3103b52-config\") pod \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"46aa86434e4345d68e5cda43f3103b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.456482 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.456380    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.456482 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.456380    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.460388 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.460372    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.460441 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.460392    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-147.us-west-1.compute.internal\": node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:05.581493 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.581437    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:05.675550 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.675515    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.677702 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:05.677685    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:05.681686 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.681665    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:05.782431 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.782405    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:05.882542 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.882494    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:05.982767 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:05.982745    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:06.078040 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.078021    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:49:06.078440 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:06.078163    2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa7caa7bfd2874950af3c2a870e3e248-6e00b72b7f2ae867.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-147.us-west-1.compute.internal.18e36d051236e67e\": read tcp 10.0.141.147:40124->3.147.98.35:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-147.us-west-1.compute.internal.18e36d051236e67e  default   4804 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-147.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-147.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-147.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:05 +0000 UTC,LastTimestamp:2026-10-30 21:49:05.334430835 +0000 UTC m=+0.544553238,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-147.us-west-1.compute.internal,}"
Oct 30 21:49:06.083315 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:06.083286    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:06.184052 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:06.184004    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:06.284525 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:06.284501    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:06.361675 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.361655    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:06.362659 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.362635    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:06.362758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.362672    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:06.362758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.362688    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:06.373294 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.373264    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:06 +0000 UTC" deadline="2028-05-28 03:36:44.908319 +0000 UTC"
Oct 30 21:49:06.373294 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.373289    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13805h47m38.535032837s"
Oct 30 21:49:06.385421 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:06.385400    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:06.485895 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:06.485835    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:06.537730 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.537703    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:06.550400 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:06.550380    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:06.586095 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:06.586073    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:07.813832 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:07.813803    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:07.846165 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:07.846128    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:07.881440 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:07.881391    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:07.925665 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:07.925640    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2135f0c5d9f6a303bc10153d13e438d.slice/crio-5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4 WatchSource:0}: Error finding container 5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4: Status 404 returned error can't find the container with id 5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4
Oct 30 21:49:07.925894 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:07.925880    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aa86434e4345d68e5cda43f3103b52.slice/crio-2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36 WatchSource:0}: Error finding container 2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36: Status 404 returned error can't find the container with id 2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36
Oct 30 21:49:07.928685 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:07.928669    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:07.931502 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:07.931489    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:49:07.949156 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:07.949140    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:07.949925 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:07.949913    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:08.006325 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.006305    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:08.254501 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.254435    2572 apiserver.go:52] "Watching apiserver"
Oct 30 21:49:08.280601 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.280582    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:08.359111 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.359087    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:49:08.359418 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.359402    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-h8h85","openshift-network-diagnostics/network-check-target-cjd7c","openshift-multus/multus-xs8hn","kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal","openshift-image-registry/node-ca-f2zxw","openshift-ovn-kubernetes/ovnkube-node-8h6kj","kube-system/global-pull-secret-syncer-jl8q2","kube-system/konnectivity-agent-d77b4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws","openshift-dns/node-resolver-q5tqg","openshift-multus/multus-additional-cni-plugins-dwxtj","openshift-network-operator/iptables-alerter-29pdk","openshift-cluster-node-tuning-operator/tuned-rcxbn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"]
Oct 30 21:49:08.363220 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.363186    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.365305 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.365288    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.367878 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.367859    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.369940 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.369923    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.372071 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.372056    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.374558 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.374539    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.376841 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.376824    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.378869 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.378856    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.378934 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.378914    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:08.381083 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.381067    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:08.381153 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.381135    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:08.384825 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.384808    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:08.384888 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.384863    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:08.387195 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.387177    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.389457 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.389425    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"e2135f0c5d9f6a303bc10153d13e438d","Type":"ContainerStarted","Data":"5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4"}
Oct 30 21:49:08.389526 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.389471    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"46aa86434e4345d68e5cda43f3103b52","Type":"ContainerStarted","Data":"2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36"}
Oct 30 21:49:08.389526 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.389510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.394023 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzbf\" (UniqueName: \"kubernetes.io/projected/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-kube-api-access-gfzbf\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394109 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394033    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-netd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.394109 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394055    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-var-lib-kubelet\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.394109 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394077    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-tuned\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.394281 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394128    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-multus\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394281 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394150    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-kubelet\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.394281 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394181    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.394281 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9k76\" (UniqueName: \"kubernetes.io/projected/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kube-api-access-n9k76\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.394281 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394263    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-conf-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394487 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394312    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4572a1ca-c58c-4207-a080-7dae7263882a-hosts-file\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.394487 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394352    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-tmp\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.394487 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394487 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-bin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394487 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-kubelet\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394652 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394520    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-etc-kubernetes\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394652 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcde6ed5-cbad-482d-9f42-b6218e390bd2-host\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.394652 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-netns\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394652 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8vz\" (UniqueName: \"kubernetes.io/projected/bcde6ed5-cbad-482d-9f42-b6218e390bd2-kube-api-access-lj8vz\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.394781 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-dbus\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.394781 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-host\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.394781 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394739    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ae2e290-c982-4b8a-a815-d3babf2679ba-agent-certs\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.394781 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-kubelet-config\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.394923 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.394923 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394850    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.394923 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-daemon-config\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.394923 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-log-socket\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395038 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-conf\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.395038 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394971    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-sys\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.395038 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.394989    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2e290-c982-4b8a-a815-d3babf2679ba-konnectivity-ca\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.395038 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cni-binary-copy\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.395184 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395065    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-k8s-cni-cncf-io\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.395184 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-multus-certs\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.395184 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395116    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-systemd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395184 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395131    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395184 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395152    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-bin\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395184 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395178    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4572a1ca-c58c-4207-a080-7dae7263882a-tmp-dir\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.395368 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395220    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ct75\" (UniqueName: \"kubernetes.io/projected/4572a1ca-c58c-4207-a080-7dae7263882a-kube-api-access-5ct75\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.395368 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395263    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bcde6ed5-cbad-482d-9f42-b6218e390bd2-serviceca\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.395368 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-var-lib-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395368 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395302    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-config\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395368 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395319    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-env-overrides\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395365    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.395505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395404    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:08.395505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/99936bc2-3cbd-445f-9a02-f387cedc1438-kube-api-access-9tww6\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:08.395505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-modprobe-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.395505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.395639 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395639 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395568    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395639 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395608    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hr9\" (UniqueName: \"kubernetes.io/projected/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-kube-api-access-r4hr9\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395639 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395629    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:08.395755 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395660    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-systemd\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.395755 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395677    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-lib-modules\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.395755 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395695    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.395755 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.395879 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395776    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-kubernetes\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.395879 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395821    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-socket-dir-parent\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.395879 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395869    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-slash\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395988 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-node-log\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395988 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395910    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-etc-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395988 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395938    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-ovn\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.395988 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-script-lib\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.396153 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.395996    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysconfig\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.396153 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396042    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcbg\" (UniqueName: \"kubernetes.io/projected/5a6789d1-ca4c-4efe-b973-8c764ae03b89-kube-api-access-tlcbg\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.396153 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.396153 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-netns\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.396153 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396132    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-run\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.396342 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cnibin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.396342 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396171    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-hostroot\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.396342 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396195    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-systemd-units\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.396342 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.396342 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.396286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-system-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.417344 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.417323    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pqcb5\""
Oct 30 21:49:08.417621 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.417592    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:49:08.417621 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.417613    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:49:08.417929 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.417913    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:08.417929 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.417922    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mdmg\""
Oct 30 21:49:08.418062 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.417935    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:08.420379 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.420362    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:49:08.420475 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.420461    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9pjm\""
Oct 30 21:49:08.420562 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.420548    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:49:08.420616 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.420570    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:49:08.420616 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.420570    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:49:08.422201 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.422187    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:49:08.422456 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.422444    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5s29\""
Oct 30 21:49:08.422806 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.422795    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:49:08.424869 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.424856    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:49:08.424918 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.424907    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:49:08.465512 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.465491    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k97q2\""
Oct 30 21:49:08.465586 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.465571    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:49:08.465623 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.465588    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:49:08.465623 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.465593    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:49:08.465623 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.465619    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:49:08.467354 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.467338    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhgs\""
Oct 30 21:49:08.472051 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.472037    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:49:08.472107 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.472059    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:49:08.472327 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.472312    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:49:08.472382 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.472340    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:49:08.472382 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.472361    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rrcm9\""
Oct 30 21:49:08.472382 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.472374    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:49:08.472382 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.472364    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:08.474366 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.474353    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:49:08.474830 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.474818    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:08.482255 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.482242    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:49:08.497144 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497119    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-slash\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497144 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-node-log\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497156    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-etc-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497171    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-ovn\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-script-lib\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysconfig\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcbg\" (UniqueName: \"kubernetes.io/projected/5a6789d1-ca4c-4efe-b973-8c764ae03b89-kube-api-access-tlcbg\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497252    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-etc-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497252    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-node-log\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497278    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-slash\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497302    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysconfig\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.497308 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497308    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-netns\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497311    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-ovn\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-run\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497337    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-netns\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497366    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cnibin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497396    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-hostroot\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497414    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-systemd-units\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497419    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cnibin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497464    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-system-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497465    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-hostroot\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497469    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-run\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzbf\" (UniqueName: \"kubernetes.io/projected/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-kube-api-access-gfzbf\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497495    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497537    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-systemd-units\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497572    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-system-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.497766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497573    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-netd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-var-lib-kubelet\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497643    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-tuned\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497676    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-multus\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497591    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-netd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497704    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-var-lib-kubelet\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497724    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-multus\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497726    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497744    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-script-lib\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfjj\" (UniqueName: \"kubernetes.io/projected/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-kube-api-access-czfjj\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497819    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-kubelet\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497840    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-kubelet\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9k76\" (UniqueName: \"kubernetes.io/projected/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kube-api-access-n9k76\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497875    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-conf-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497898    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-conf-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.498582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cnibin\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497935    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497952    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497988    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4572a1ca-c58c-4207-a080-7dae7263882a-hosts-file\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.497959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4572a1ca-c58c-4207-a080-7dae7263882a-hosts-file\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-tmp\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498041    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-bin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-kubelet\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498093    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-etc-kubernetes\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498113    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcde6ed5-cbad-482d-9f42-b6218e390bd2-host\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498115    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-bin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-netns\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498133    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-kubelet\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498146    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcde6ed5-cbad-482d-9f42-b6218e390bd2-host\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498146    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-etc-kubernetes\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498155    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-system-cni-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498166    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-netns\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.499425 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498174    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498186    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268dd18a-b222-4c6d-b981-6babbe21878f-iptables-alerter-script\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498222    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8vz\" (UniqueName: \"kubernetes.io/projected/bcde6ed5-cbad-482d-9f42-b6218e390bd2-kube-api-access-lj8vz\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-dbus\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-host\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ae2e290-c982-4b8a-a815-d3babf2679ba-agent-certs\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498296    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-dbus\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-kubelet-config\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498331    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498334    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-host\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498355    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-daemon-config\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498387    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-kubelet-config\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498398    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-log-socket\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498421    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-conf\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498436    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-sys\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498467    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2e290-c982-4b8a-a815-d3babf2679ba-konnectivity-ca\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.500261 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.498480    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498491    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cni-binary-copy\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-k8s-cni-cncf-io\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498521    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-sys\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.498562    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret podName:2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:08.998524157 +0000 UTC m=+3.081353975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret") pod "global-pull-secret-syncer-jl8q2" (UID: "2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498596    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-k8s-cni-cncf-io\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498601    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-multus-certs\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498482    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-log-socket\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498642    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-conf\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498644    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-multus-certs\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498664    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268dd18a-b222-4c6d-b981-6babbe21878f-host-slash\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498706    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-systemd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498733    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498751    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-systemd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-bin\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4572a1ca-c58c-4207-a080-7dae7263882a-tmp-dir\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498807    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501043 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ct75\" (UniqueName: \"kubernetes.io/projected/4572a1ca-c58c-4207-a080-7dae7263882a-kube-api-access-5ct75\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bcde6ed5-cbad-482d-9f42-b6218e390bd2-serviceca\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498881    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-var-lib-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498885    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-daemon-config\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-config\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-env-overrides\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498970    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499024    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/99936bc2-3cbd-445f-9a02-f387cedc1438-kube-api-access-9tww6\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.498789    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-bin\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499047    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-modprobe-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499069    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499087    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-var-lib-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499092    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499128    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499131    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.501803 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hr9\" (UniqueName: \"kubernetes.io/projected/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-kube-api-access-r4hr9\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499280    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-systemd\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499389    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-env-overrides\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499407    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-lib-modules\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499435    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-kubernetes\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499499    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-socket-dir-parent\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.499527    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499537    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkx6\" (UniqueName: \"kubernetes.io/projected/268dd18a-b222-4c6d-b981-6babbe21878f-kube-api-access-ltkx6\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499597    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-modprobe-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499628    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-systemd\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.499655    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:08.999567152 +0000 UTC m=+3.082396958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499676    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499711    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-lib-modules\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502283 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499823    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499844    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-config\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499885    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499912    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-kubernetes\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.499939    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-socket-dir-parent\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.500168    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4572a1ca-c58c-4207-a080-7dae7263882a-tmp-dir\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.500185    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bcde6ed5-cbad-482d-9f42-b6218e390bd2-serviceca\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.500325    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cni-binary-copy\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.500395    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2e290-c982-4b8a-a815-d3babf2679ba-konnectivity-ca\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.501947    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-tmp\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.502047    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ae2e290-c982-4b8a-a815-d3babf2679ba-agent-certs\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.502638    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-tuned\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.502758 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.502718    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.515408 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.515352    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wvpps\""
Oct 30 21:49:08.515408 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.515363    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs82w\""
Oct 30 21:49:08.515408 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.515372    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:49:08.515408 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.515397    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:49:08.515610 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.515400    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:49:08.515610 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.515520    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:49:08.515610 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.515546    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mghjq"
Oct 30 21:49:08.559975 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.559955    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:08.560071 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.559979    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:08.560116 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.560068    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pc6rv for pod openshift-network-diagnostics/network-check-target-cjd7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:08.560244 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:08.560230    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv podName:e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:09.060194458 +0000 UTC m=+3.143024287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pc6rv" (UniqueName: "kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv") pod "network-check-target-cjd7c" (UID: "e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:08.561536 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.561517    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcbg\" (UniqueName: \"kubernetes.io/projected/5a6789d1-ca4c-4efe-b973-8c764ae03b89-kube-api-access-tlcbg\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.561609 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.561586    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzbf\" (UniqueName: \"kubernetes.io/projected/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-kube-api-access-gfzbf\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.562127 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.562109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9k76\" (UniqueName: \"kubernetes.io/projected/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kube-api-access-n9k76\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.562173 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.562109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8vz\" (UniqueName: \"kubernetes.io/projected/bcde6ed5-cbad-482d-9f42-b6218e390bd2-kube-api-access-lj8vz\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.566472 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.566449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/99936bc2-3cbd-445f-9a02-f387cedc1438-kube-api-access-9tww6\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:08.566790 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.566771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hr9\" (UniqueName: \"kubernetes.io/projected/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-kube-api-access-r4hr9\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.566824 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.566772    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ct75\" (UniqueName: \"kubernetes.io/projected/4572a1ca-c58c-4207-a080-7dae7263882a-kube-api-access-5ct75\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.600785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600763    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.600856 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfjj\" (UniqueName: \"kubernetes.io/projected/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-kube-api-access-czfjj\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.600856 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600806    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cnibin\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.600856 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.600856 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-system-cni-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600872    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268dd18a-b222-4c6d-b981-6babbe21878f-iptables-alerter-script\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600850    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cnibin\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600899    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-system-cni-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600925    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268dd18a-b222-4c6d-b981-6babbe21878f-host-slash\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600986    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600998    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkx6\" (UniqueName: \"kubernetes.io/projected/268dd18a-b222-4c6d-b981-6babbe21878f-kube-api-access-ltkx6\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.601044 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.600999    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268dd18a-b222-4c6d-b981-6babbe21878f-host-slash\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.601323 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.601053    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.601996 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.601976    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.602067 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.602025    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.602067 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.602057    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268dd18a-b222-4c6d-b981-6babbe21878f-iptables-alerter-script\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.602155 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.602092    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.662818 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.662793    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfjj\" (UniqueName: \"kubernetes.io/projected/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-kube-api-access-czfjj\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.662906 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.662869    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkx6\" (UniqueName: \"kubernetes.io/projected/268dd18a-b222-4c6d-b981-6babbe21878f-kube-api-access-ltkx6\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.673801 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.673784    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:08.680722 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.680701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:08.683040 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.683018    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4572a1ca_c58c_4207_a080_7dae7263882a.slice/crio-75da86808613f6af8b190bb7c1afd414b89634718109eda5e8bef1d3b66203c1 WatchSource:0}: Error finding container 75da86808613f6af8b190bb7c1afd414b89634718109eda5e8bef1d3b66203c1: Status 404 returned error can't find the container with id 75da86808613f6af8b190bb7c1afd414b89634718109eda5e8bef1d3b66203c1
Oct 30 21:49:08.687545 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.687386    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:08.689344 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.689324    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6789d1_ca4c_4efe_b973_8c764ae03b89.slice/crio-432fe6d8dc2f88f3378f0aa7328728a27d4ecb883add14b24b8000364a8939e0 WatchSource:0}: Error finding container 432fe6d8dc2f88f3378f0aa7328728a27d4ecb883add14b24b8000364a8939e0: Status 404 returned error can't find the container with id 432fe6d8dc2f88f3378f0aa7328728a27d4ecb883add14b24b8000364a8939e0
Oct 30 21:49:08.691405 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.691387    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:08.697071 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.697046    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf66beb_ad8b_47f2_89cc_9b8e6bc18b82.slice/crio-5ebb8f08c89cc3e2a841eef810f0ecfbff47a7d4487cb6075cac2b569fb0b3a8 WatchSource:0}: Error finding container 5ebb8f08c89cc3e2a841eef810f0ecfbff47a7d4487cb6075cac2b569fb0b3a8: Status 404 returned error can't find the container with id 5ebb8f08c89cc3e2a841eef810f0ecfbff47a7d4487cb6075cac2b569fb0b3a8
Oct 30 21:49:08.697071 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.697060    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:08.699628 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.699608    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae2e290_c982_4b8a_a815_d3babf2679ba.slice/crio-4dcc139cffd7b6c9b5f1e3d558d85a519322c05cde86d44a84cdbb44e2fa21e4 WatchSource:0}: Error finding container 4dcc139cffd7b6c9b5f1e3d558d85a519322c05cde86d44a84cdbb44e2fa21e4: Status 404 returned error can't find the container with id 4dcc139cffd7b6c9b5f1e3d558d85a519322c05cde86d44a84cdbb44e2fa21e4
Oct 30 21:49:08.706325 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.706297    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcde6ed5_cbad_482d_9f42_b6218e390bd2.slice/crio-2bfdbbadcc4984a12e470900f67e8c5b696bbd60975f278a5a16876aa76d3217 WatchSource:0}: Error finding container 2bfdbbadcc4984a12e470900f67e8c5b696bbd60975f278a5a16876aa76d3217: Status 404 returned error can't find the container with id 2bfdbbadcc4984a12e470900f67e8c5b696bbd60975f278a5a16876aa76d3217
Oct 30 21:49:08.722358 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.722339    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:08.728041 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.728027    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:08.730538 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.730518    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:08.730713 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.730672    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8cd4cd_5b2d_400f_b3b0_f84abc42bcf1.slice/crio-4aa635da86fa6866138ac0a2957518a7984ddd913ef3e1c5fe487c13ef15ccf8 WatchSource:0}: Error finding container 4aa635da86fa6866138ac0a2957518a7984ddd913ef3e1c5fe487c13ef15ccf8: Status 404 returned error can't find the container with id 4aa635da86fa6866138ac0a2957518a7984ddd913ef3e1c5fe487c13ef15ccf8
Oct 30 21:49:08.738405 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.738387    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268dd18a_b222_4c6d_b981_6babbe21878f.slice/crio-55e1f0b2c6b61cd08c854054a7562c94b1d1a8a8b042ac432a7e876805dcb5c2 WatchSource:0}: Error finding container 55e1f0b2c6b61cd08c854054a7562c94b1d1a8a8b042ac432a7e876805dcb5c2: Status 404 returned error can't find the container with id 55e1f0b2c6b61cd08c854054a7562c94b1d1a8a8b042ac432a7e876805dcb5c2
Oct 30 21:49:08.740410 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:08.740397    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:08.740934 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.740918    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bbb59a_b1e7_4993_8a0b_b82f54b1c844.slice/crio-2587c6a50a557f5c522cb7d8c1589baba2d4e84bee219796d490ee6b2c4b23bd WatchSource:0}: Error finding container 2587c6a50a557f5c522cb7d8c1589baba2d4e84bee219796d490ee6b2c4b23bd: Status 404 returned error can't find the container with id 2587c6a50a557f5c522cb7d8c1589baba2d4e84bee219796d490ee6b2c4b23bd
Oct 30 21:49:08.748641 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:08.748620    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d6e046_0ec6_4b8a_bf36_c43a3abd5ba4.slice/crio-84e04ea362d6752c70c4ac2f6b3740f33e5d9570ae17d399ee9246731b4fddc0 WatchSource:0}: Error finding container 84e04ea362d6752c70c4ac2f6b3740f33e5d9570ae17d399ee9246731b4fddc0: Status 404 returned error can't find the container with id 84e04ea362d6752c70c4ac2f6b3740f33e5d9570ae17d399ee9246731b4fddc0
Oct 30 21:49:09.005362 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.005275    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:09.005362 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.005355    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:09.005578 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.005433    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:09.005578 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.005472    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:09.005578 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.005510    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret podName:2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:10.005487646 +0000 UTC m=+4.088317461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret") pod "global-pull-secret-syncer-jl8q2" (UID: "2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:09.005578 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.005530    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:10.00552039 +0000 UTC m=+4.088350192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:09.106667 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.106632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:09.106856 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.106759    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:09.106856 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.106776    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:09.106856 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.106789    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pc6rv for pod openshift-network-diagnostics/network-check-target-cjd7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:09.106856 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:09.106845    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv podName:e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:10.106831201 +0000 UTC m=+4.189661002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc6rv" (UniqueName: "kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv") pod "network-check-target-cjd7c" (UID: "e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:09.365931 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.365835    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29pdk" event={"ID":"268dd18a-b222-4c6d-b981-6babbe21878f","Type":"ContainerStarted","Data":"55e1f0b2c6b61cd08c854054a7562c94b1d1a8a8b042ac432a7e876805dcb5c2"}
Oct 30 21:49:09.367511 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.367472    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws" event={"ID":"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1","Type":"ContainerStarted","Data":"4aa635da86fa6866138ac0a2957518a7984ddd913ef3e1c5fe487c13ef15ccf8"}
Oct 30 21:49:09.368945 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.368917    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d77b4" event={"ID":"7ae2e290-c982-4b8a-a815-d3babf2679ba","Type":"ContainerStarted","Data":"4dcc139cffd7b6c9b5f1e3d558d85a519322c05cde86d44a84cdbb44e2fa21e4"}
Oct 30 21:49:09.370343 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.370322    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcxbn" event={"ID":"5a6789d1-ca4c-4efe-b973-8c764ae03b89","Type":"ContainerStarted","Data":"432fe6d8dc2f88f3378f0aa7328728a27d4ecb883add14b24b8000364a8939e0"}
Oct 30 21:49:09.371998 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.371973    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5tqg" event={"ID":"4572a1ca-c58c-4207-a080-7dae7263882a","Type":"ContainerStarted","Data":"75da86808613f6af8b190bb7c1afd414b89634718109eda5e8bef1d3b66203c1"}
Oct 30 21:49:09.373630 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.373607    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerStarted","Data":"84e04ea362d6752c70c4ac2f6b3740f33e5d9570ae17d399ee9246731b4fddc0"}
Oct 30 21:49:09.375148 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.375126    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"2587c6a50a557f5c522cb7d8c1589baba2d4e84bee219796d490ee6b2c4b23bd"}
Oct 30 21:49:09.376512 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.376465    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f2zxw" event={"ID":"bcde6ed5-cbad-482d-9f42-b6218e390bd2","Type":"ContainerStarted","Data":"2bfdbbadcc4984a12e470900f67e8c5b696bbd60975f278a5a16876aa76d3217"}
Oct 30 21:49:09.377695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.377664    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xs8hn" event={"ID":"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82","Type":"ContainerStarted","Data":"5ebb8f08c89cc3e2a841eef810f0ecfbff47a7d4487cb6075cac2b569fb0b3a8"}
Oct 30 21:49:09.519817 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.519772    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:08 +0000 UTC" deadline="2028-07-30 05:50:23.365484344 +0000 UTC"
Oct 30 21:49:09.519817 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:09.519814    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15320h1m13.845676668s"
Oct 30 21:49:10.013774 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.013739    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:10.013912 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.013811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:10.013987 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.013950    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:10.014041 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.014005    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret podName:2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.013988098 +0000 UTC m=+6.096817904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret") pod "global-pull-secret-syncer-jl8q2" (UID: "2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:10.014167 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.014066    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:10.014167 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.014098    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.014086886 +0000 UTC m=+6.096916688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:10.114786 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.114462    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:10.114786 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.114667    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:10.114786 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.114687    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:10.114786 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.114698    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pc6rv for pod openshift-network-diagnostics/network-check-target-cjd7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:10.114786 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.114752    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv podName:e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.114734988 +0000 UTC m=+6.197564793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc6rv" (UniqueName: "kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv") pod "network-check-target-cjd7c" (UID: "e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:10.360746 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.360509    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:10.360746 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.360519    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:10.360746 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.360656    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:10.360746 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.360670    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:10.361037 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.360776    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:10.361037 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:10.360875    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:10.520112 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.520048    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:08 +0000 UTC" deadline="2028-04-13 03:43:05.35709376 +0000 UTC"
Oct 30 21:49:10.520112 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:10.520109    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12725h53m54.836988985s"
Oct 30 21:49:11.407645 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:11.407312    2572 generic.go:356] "Generic (PLEG): container finished" podID="e2135f0c5d9f6a303bc10153d13e438d" containerID="009bf78a94203c34d2031d5015b0a6df28756d15b7e78929ff2273f329a997e4" exitCode=0
Oct 30 21:49:11.407645 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:11.407368    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"e2135f0c5d9f6a303bc10153d13e438d","Type":"ContainerDied","Data":"009bf78a94203c34d2031d5015b0a6df28756d15b7e78929ff2273f329a997e4"}
Oct 30 21:49:12.027881 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.027502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:12.027881 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.027571    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:12.027881 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.027693    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:12.027881 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.027748    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret podName:2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:16.027734502 +0000 UTC m=+10.110564316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret") pod "global-pull-secret-syncer-jl8q2" (UID: "2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:12.027881 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.027793    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:12.027881 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.027815    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:16.027807563 +0000 UTC m=+10.110637364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:12.128572 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.128535    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:12.128724 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.128711    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:12.128846 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.128729    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:12.128846 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.128742    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pc6rv for pod openshift-network-diagnostics/network-check-target-cjd7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:12.128846 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.128798    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv podName:e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:16.128780389 +0000 UTC m=+10.211610205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc6rv" (UniqueName: "kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv") pod "network-check-target-cjd7c" (UID: "e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:12.359806 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.359731    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:12.359958 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.359830    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:12.359958 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.359872    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:12.359958 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.359935    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:12.360105 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.359975    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:12.360105 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:12.360023    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:12.418678 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.418641    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"e2135f0c5d9f6a303bc10153d13e438d","Type":"ContainerStarted","Data":"7a6c237ebfc373c7799e27fc55c55b2ca8aaf35ba0027ebc8ebb3c0542d6bd3d"}
Oct 30 21:49:12.421787 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.421760    2572 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="b1a5035a137b2a4474009d2a6e2892d49395279a6024f81c08947624bd5712f4" exitCode=0
Oct 30 21:49:12.421905 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.421849    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerDied","Data":"b1a5035a137b2a4474009d2a6e2892d49395279a6024f81c08947624bd5712f4"}
Oct 30 21:49:12.434755 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.434729    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"46aa86434e4345d68e5cda43f3103b52","Type":"ContainerStarted","Data":"9101a98786086577667bf20840da1426d13d0f6cf5d3dd1b936e84f8c21c34c8"}
Oct 30 21:49:12.538635 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:12.538148    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal" podStartSLOduration=5.538129451 podStartE2EDuration="5.538129451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:12.536917164 +0000 UTC m=+6.619746987" watchObservedRunningTime="2026-10-30 21:49:12.538129451 +0000 UTC m=+6.620959275"
Oct 30 21:49:13.446771 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:13.446693    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d77b4" event={"ID":"7ae2e290-c982-4b8a-a815-d3babf2679ba","Type":"ContainerStarted","Data":"9a42f7c90260faa227c49cd07bc420622586b4de1a4b0e57f000ed4c4bc699e2"}
Oct 30 21:49:13.564218 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:13.563532    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal" podStartSLOduration=6.563517429 podStartE2EDuration="6.563517429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:12.780503651 +0000 UTC m=+6.863333474" watchObservedRunningTime="2026-10-30 21:49:13.563517429 +0000 UTC m=+7.646347251"
Oct 30 21:49:14.360546 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:14.360039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:14.360546 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:14.360162    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:14.360546 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:14.360258    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:14.360546 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:14.360341    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:14.360546 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:14.360394    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:14.360546 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:14.360455    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:14.453414 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:14.453307    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f2zxw" event={"ID":"bcde6ed5-cbad-482d-9f42-b6218e390bd2","Type":"ContainerStarted","Data":"e9ca0a02ca1ba2b90d602cea9a6327d92a947eeed358aa69b964328c5a218dd7"}
Oct 30 21:49:14.569592 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:14.569273    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d77b4" podStartSLOduration=4.072467085 podStartE2EDuration="8.569259034s" totalImagesPullingTime="4.496791949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:13.564111349 +0000 UTC m=+7.646941171" watchObservedRunningTime="2026-10-30 21:49:14.569259034 +0000 UTC m=+8.652088858"
Oct 30 21:49:15.461326 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:15.461187    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws" event={"ID":"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1","Type":"ContainerStarted","Data":"c0b987da2397236b511c28d0ca225ce4b36be4c07f69e9a5214ff01904bb2d37"}
Oct 30 21:49:16.058519 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:16.058473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:16.058701 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:16.058545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:16.058701 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.058685    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:16.058800 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.058746    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.058730079 +0000 UTC m=+18.141559898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:16.058854 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.058805    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:16.058854 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.058832    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret podName:2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.058823393 +0000 UTC m=+18.141653206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret") pod "global-pull-secret-syncer-jl8q2" (UID: "2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:16.159446 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:16.159411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:16.159614 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.159594    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:16.159614 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.159611    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:16.159724 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.159621    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pc6rv for pod openshift-network-diagnostics/network-check-target-cjd7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:16.159724 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.159662    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv podName:e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.159650008 +0000 UTC m=+18.242479813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc6rv" (UniqueName: "kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv") pod "network-check-target-cjd7c" (UID: "e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:16.360493 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:16.359953    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:16.360493 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:16.359991    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:16.360493 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.360071    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:16.360493 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:16.359956    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:16.360493 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.360172    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:16.360493 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:16.360281    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:17.060182 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.059574    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:17.060597 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.060444    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:17.177430 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.177380    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f2zxw" podStartSLOduration=5.767687558 podStartE2EDuration="11.177362299s" totalImagesPullingTime="5.409674741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:14.569576245 +0000 UTC m=+8.652406067" watchObservedRunningTime="2026-10-30 21:49:17.177362299 +0000 UTC m=+11.260192121"
Oct 30 21:49:17.212927 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.212905    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:49:17.379185 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.379078    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:49:17.21292266Z","UUID":"e1ae80df-c00b-4557-a640-eff2a4f70a39","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:49:17.381122 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.381102    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:49:17.381258 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.381130    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:49:17.468695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.468619    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws" event={"ID":"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1","Type":"ContainerStarted","Data":"96ca1e97bd10858e01015e73b4712adfe8095f95fc35f0cc33df72f2d0e51feb"}
Oct 30 21:49:17.470932 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.470901    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5tqg" event={"ID":"4572a1ca-c58c-4207-a080-7dae7263882a","Type":"ContainerStarted","Data":"aadd49cd158e8d4839e7896fa985867019a8fa078b0636bbf143c350868de424"}
Oct 30 21:49:17.473149 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.473119    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerStarted","Data":"531e04af24a5f0a0aa8658b765902260bb3e8dd6ffa6736d2fad0680957d3812"}
Oct 30 21:49:17.476006 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.475979    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xs8hn" event={"ID":"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82","Type":"ContainerStarted","Data":"91e3923a9b58b6437036e16ed28a69bd2ac2c7a3b964dd387217a2eb7c31fc33"}
Oct 30 21:49:17.590739 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.590632    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q5tqg" podStartSLOduration=3.189249475 podStartE2EDuration="11.590612924s" totalImagesPullingTime="8.401363449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:17.590456224 +0000 UTC m=+11.673286055" watchObservedRunningTime="2026-10-30 21:49:17.590612924 +0000 UTC m=+11.673442748"
Oct 30 21:49:17.710431 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:17.710371    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xs8hn" podStartSLOduration=3.22849228 podStartE2EDuration="11.710357305s" totalImagesPullingTime="8.481865025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:17.710157644 +0000 UTC m=+11.792987478" watchObservedRunningTime="2026-10-30 21:49:17.710357305 +0000 UTC m=+11.793187127"
Oct 30 21:49:18.082854 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:18.082772    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:18.083479 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:18.083454    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:18.359407 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:18.359333    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:18.359554 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:18.359446    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:18.359616 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:18.359457    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:18.359668 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:18.359620    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:18.359727 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:18.359704    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:18.359809 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:18.359783    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:18.480388 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:18.480348    2572 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="531e04af24a5f0a0aa8658b765902260bb3e8dd6ffa6736d2fad0680957d3812" exitCode=0
Oct 30 21:49:18.481440 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:18.481406    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerDied","Data":"531e04af24a5f0a0aa8658b765902260bb3e8dd6ffa6736d2fad0680957d3812"}
Oct 30 21:49:19.484958 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:19.484927    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29pdk" event={"ID":"268dd18a-b222-4c6d-b981-6babbe21878f","Type":"ContainerStarted","Data":"5eebd6810319d023a6c9046efd98e9ef69f318f3e1742f4d3227ea81075cfcae"}
Oct 30 21:49:19.487836 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:19.487772    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws" event={"ID":"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1","Type":"ContainerStarted","Data":"58e45035fd021b7f18bcc907e82688958aeb314082aad1bae546a05a13ad3e6c"}
Oct 30 21:49:19.489721 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:19.489695    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcxbn" event={"ID":"5a6789d1-ca4c-4efe-b973-8c764ae03b89","Type":"ContainerStarted","Data":"f5cc8ef7735089abb0ba03ef7037b924229e3e1b7f93f35823d83c2a4841de5c"}
Oct 30 21:49:19.604943 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:19.604887    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-29pdk" podStartSLOduration=5.382778915 podStartE2EDuration="13.604870362s" totalImagesPullingTime="8.222091447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:19.604379267 +0000 UTC m=+13.687209090" watchObservedRunningTime="2026-10-30 21:49:19.604870362 +0000 UTC m=+13.687700188"
Oct 30 21:49:19.727113 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:19.727019    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws" podStartSLOduration=3.8183467760000003 podStartE2EDuration="13.727003424s" totalImagesPullingTime="9.908656648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:19.726836127 +0000 UTC m=+13.809665949" watchObservedRunningTime="2026-10-30 21:49:19.727003424 +0000 UTC m=+13.809833248"
Oct 30 21:49:19.857311 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:19.857257    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rcxbn" podStartSLOduration=3.148295058 podStartE2EDuration="13.857236393s" totalImagesPullingTime="10.708941335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:19.851922266 +0000 UTC m=+13.934752113" watchObservedRunningTime="2026-10-30 21:49:19.857236393 +0000 UTC m=+13.940066218"
Oct 30 21:49:20.358030 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:20.357994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:20.358188 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:20.358105    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:20.358507 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:20.358491    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:20.358607 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:20.358590    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:20.358732 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:20.358716    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:20.358804 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:20.358789    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:20.493636 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:20.493604    2572 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="74b0ea871ccf0614a6e56e46dfce62525c929732448a46f8c7371ef1f180cf1d" exitCode=0
Oct 30 21:49:20.494188 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:20.493715    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerDied","Data":"74b0ea871ccf0614a6e56e46dfce62525c929732448a46f8c7371ef1f180cf1d"}
Oct 30 21:49:22.357981 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:22.357955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:22.358797 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:22.357949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:22.358797 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:22.358078    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:22.358797 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:22.358146    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:22.358797 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:22.357956    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:22.358797 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:22.358250    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:22.500674 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:22.500645    2572 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="928c4206450b07636e91ea3fed57ceafacdb27f354fb5910fbc7dc4647dafc7a" exitCode=0
Oct 30 21:49:22.500865 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:22.500690    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerDied","Data":"928c4206450b07636e91ea3fed57ceafacdb27f354fb5910fbc7dc4647dafc7a"}
Oct 30 21:49:24.120170 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.120006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:24.120556 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.120159    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:24.120556 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.120243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:24.120556 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.120297    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret podName:2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:40.120271363 +0000 UTC m=+34.203101177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret") pod "global-pull-secret-syncer-jl8q2" (UID: "2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:24.120556 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.120349    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:24.120556 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.120397    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:40.120384554 +0000 UTC m=+34.203214356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:24.221387 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.221361    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:24.221524 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.221506    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:24.221585 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.221532    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:24.221585 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.221545    2572 projected.go:196] Error preparing data for projected volume kube-api-access-pc6rv for pod openshift-network-diagnostics/network-check-target-cjd7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:24.221661 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.221603    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv podName:e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:40.221584531 +0000 UTC m=+34.304414335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc6rv" (UniqueName: "kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv") pod "network-check-target-cjd7c" (UID: "e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:24.357944 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.357918    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:24.358102 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.357960    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:24.358102 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.357963    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:24.358102 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.358069    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:24.358247 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.358153    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:24.358284 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:24.358267    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:24.507372 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.507345    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"47d974f028b0656e2bd2c8b1a08f40125201767ad2ae0b926e1946cf4b50585b"}
Oct 30 21:49:24.507484 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.507455    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"ee0600d099f1bf42c8ff066dce7b8b7427328c16d4d2ef8b98cd7c76039c3546"}
Oct 30 21:49:24.507556 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.507500    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"c0a76bcbe3eefb9381d76760c7ff3b65da1d15278ae2387a164e14beeeaa743b"}
Oct 30 21:49:24.507556 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.507530    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"491e72b07c8b2b9d9be6556828edf42f58e4a549a09682f282385999d1d44968"}
Oct 30 21:49:24.507651 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.507561    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"14876f52674cfcce56f742ea11079f1f6313cadd13a78f99586d064a231b3a3e"}
Oct 30 21:49:24.507651 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:24.507592    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"99666379b47198c7e1468b1aa009c1a465fc3dd2a098de3390e6ede16b43ee52"}
Oct 30 21:49:25.512328 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:25.512107    2572 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="023e0f1ad6bdc5e06c2eaeae4c2f189a0f1d94aa1fc37977aae8847c150f53bf" exitCode=0
Oct 30 21:49:25.512328 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:25.512202    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerDied","Data":"023e0f1ad6bdc5e06c2eaeae4c2f189a0f1d94aa1fc37977aae8847c150f53bf"}
Oct 30 21:49:26.359050 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:26.359026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:26.359193 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:26.359174    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:26.359291 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:26.359201    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:26.359291 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:26.359257    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:26.359394 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:26.359370    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:26.359498 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:26.359476    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:26.516972 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:26.516939    2572 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="56a5f72177009507be1b949fa056f6d008eafc0b4902a0423ae35e38fe991160" exitCode=0
Oct 30 21:49:26.517383 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:26.517020    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerDied","Data":"56a5f72177009507be1b949fa056f6d008eafc0b4902a0423ae35e38fe991160"}
Oct 30 21:49:26.520454 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:26.520435    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"5e6ad5da714b297b1884946bbe27777c60006833a54b76b1a89c5fe48e5efba6"}
Oct 30 21:49:27.525492 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:27.525456    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" event={"ID":"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4","Type":"ContainerStarted","Data":"e4b18f9b7e59670bfb211d77a9ab693e33003dd07612f689b4a4d37f9728e3f2"}
Oct 30 21:49:27.701339 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:27.701286    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dwxtj" podStartSLOduration=9.207701366 podStartE2EDuration="21.701273969s" totalImagesPullingTime="12.493572603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:27.700815573 +0000 UTC m=+21.783645408" watchObservedRunningTime="2026-10-30 21:49:27.701273969 +0000 UTC m=+21.784103792"
Oct 30 21:49:28.358406 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:28.358374    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:28.358541 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:28.358427    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:28.358541 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:28.358508    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:28.358635 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:28.358529    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:28.358635 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:28.358600    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:28.358706 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:28.358665    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:28.532533 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:28.532504    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" event={"ID":"45bbb59a-b1e7-4993-8a0b-b82f54b1c844","Type":"ContainerStarted","Data":"6f7f8078b8155457aef7d55e7392a66157f9d8134cf75a3aada9cafed3a48b16"}
Oct 30 21:49:28.533200 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:28.533020    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:28.549068 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:28.549047    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:28.798713 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:28.798623    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj" podStartSLOduration=7.594374531 podStartE2EDuration="22.79861058s" totalImagesPullingTime="15.204236049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:28.798471675 +0000 UTC m=+22.881301497" watchObservedRunningTime="2026-10-30 21:49:28.79861058 +0000 UTC m=+22.881440427"
Oct 30 21:49:29.534739 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:29.534709    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:29.535261 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:29.534779    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:29.549942 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:29.549920    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:30.357835 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:30.357802    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:30.358013 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:30.357835    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:30.358013 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:30.357898    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:30.358013 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:30.357946    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:30.358183 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:30.358043    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:30.358183 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:30.358137    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:31.069664 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:31.069635    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jl8q2"]
Oct 30 21:49:31.070053 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:31.069725    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:31.070053 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:31.069800    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:31.072730 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:31.072703    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjd7c"]
Oct 30 21:49:31.072875 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:31.072800    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:31.072934 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:31.072895    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:31.073457 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:31.073433    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h8h85"]
Oct 30 21:49:31.073555 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:31.073533    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:31.073661 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:31.073643    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:32.358074 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:32.358038    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:32.358599 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:32.358039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:32.358599 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:32.358269    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:32.358599 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:32.358141    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:33.358517 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:33.358346    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:33.358863 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:33.358601    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:34.358321 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:34.358286    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:34.358574 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:34.358376    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:34.358574 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:34.358470    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:34.359088 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:34.358572    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:35.358364 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:35.358339    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:35.358521 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:35.358441    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8h85" podUID="99936bc2-3cbd-445f-9a02-f387cedc1438"
Oct 30 21:49:36.360225 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:36.360194    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:36.360651 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:36.360306    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjd7c" podUID="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66"
Oct 30 21:49:36.360651 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:36.360360    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:36.360651 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:36.360440    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jl8q2" podUID="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5"
Oct 30 21:49:37.190429 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.190348    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:49:37.190559 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.190515    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:49:37.306990 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.306958    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:49:37.307162 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.307012    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m87n9"]
Oct 30 21:49:37.311826 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.311798    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jqx4n"]
Oct 30 21:49:37.311938 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.311909    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m87n9"
Oct 30 21:49:37.314567 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.314550    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"]
Oct 30 21:49:37.314672 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.314662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jqx4n"
Oct 30 21:49:37.317135 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.317119    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.356460 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.356429    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"]
Oct 30 21:49:37.359505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.359487    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w9w7t"]
Oct 30 21:49:37.359602 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.359589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.359647 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.359622    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:37.362600 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.362581    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59f787ddb6-57tbv"]
Oct 30 21:49:37.362876 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.362686    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w9w7t"
Oct 30 21:49:37.365071 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.365055    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xh5b2"]
Oct 30 21:49:37.365164 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.365150    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.367835 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.367819    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d48b6d977-dstqd"]
Oct 30 21:49:37.367949 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.367934    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.371489 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.371470    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"]
Oct 30 21:49:37.371578 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.371563    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.374588 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.374573    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-g64qk"]
Oct 30 21:49:37.374713 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.374696    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"
Oct 30 21:49:37.377342 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.377326    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8698788495-986ld"]
Oct 30 21:49:37.377449 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.377436    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:37.379945 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.379929    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fs49h"]
Oct 30 21:49:37.380056 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.380040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.382618 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.382602    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"]
Oct 30 21:49:37.382733 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.382718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.385306 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.385289    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"]
Oct 30 21:49:37.385505 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.385489    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.389037 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.389011    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"]
Oct 30 21:49:37.389727 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.389705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.392700 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.392682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"
Oct 30 21:49:37.405756 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.405711    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xrgwp"]
Oct 30 21:49:37.408725 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.408707    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5xrdj"]
Oct 30 21:49:37.408852 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.408833    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.412428 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.412411    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m87n9"]
Oct 30 21:49:37.412538 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.412445    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"]
Oct 30 21:49:37.412538 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.412471    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jqx4n"]
Oct 30 21:49:37.412768 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.412752    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:49:37.413000 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.412985    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.413123 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.413112    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.413236 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.413112    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.413331 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.413125    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.413393 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.413282    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.413596 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.413575    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:49:37.414201 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.414186    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:49:37.414527 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.414506    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:49:37.414753 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.414736    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.417591 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.417575    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf193224-26cc-4cd6-8aa5-2424e1212c2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.417664 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.417599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.417664 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.417627    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-472cs\" (UniqueName: \"kubernetes.io/projected/1fdc58d1-11b9-4879-8879-8909f186bbe5-kube-api-access-472cs\") pod \"kas-connection-checker-685895c7c9-jqx4n\" (UID: \"1fdc58d1-11b9-4879-8879-8909f186bbe5\") " pod="kube-system/kas-connection-checker-685895c7c9-jqx4n"
Oct 30 21:49:37.417766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.417662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmmt\" (UniqueName: \"kubernetes.io/projected/bf193224-26cc-4cd6-8aa5-2424e1212c2a-kube-api-access-7cmmt\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.417766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.417670    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:49:37.417858 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.417830    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cr2j\" (UniqueName: \"kubernetes.io/projected/920770c6-fd18-41d7-95f6-a55c195361e6-kube-api-access-6cr2j\") pod \"kas-connection-checker-685895c7c9-m87n9\" (UID: \"920770c6-fd18-41d7-95f6-a55c195361e6\") " pod="kube-system/kas-connection-checker-685895c7c9-m87n9"
Oct 30 21:49:37.419144 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.419122    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w9w7t"]
Oct 30 21:49:37.428198 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.428176    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"]
Oct 30 21:49:37.429130 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.429109    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"]
Oct 30 21:49:37.455837 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.455813    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-g64qk"]
Oct 30 21:49:37.457038 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.456852    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d48b6d977-dstqd"]
Oct 30 21:49:37.457691 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.457674    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f787ddb6-57tbv"]
Oct 30 21:49:37.458354 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.458339    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8698788495-986ld"]
Oct 30 21:49:37.475929 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.475907    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrgwp"]
Oct 30 21:49:37.481148 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.481050    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:37.486103 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.486054    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"]
Oct 30 21:49:37.487150 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.487126    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xh5b2"]
Oct 30 21:49:37.494792 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.494390    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5xrdj"]
Oct 30 21:49:37.497959 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.497935    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"]
Oct 30 21:49:37.498609 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.498586    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fs49h"]
Oct 30 21:49:37.511296 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.511277    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:49:37.511437 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.511419    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:49:37.511574 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.511558    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.511632 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.511582    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:49:37.511774 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.511745    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:49:37.512030 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512011    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:49:37.512030 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512026    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.512030 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512047    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:49:37.512194 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512062    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jm7gq\""
Oct 30 21:49:37.512293 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512262    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:49:37.512293 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512267    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.512396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512295    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.512396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512302    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vjtf6\""
Oct 30 21:49:37.512396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512307    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.512396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512312    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.512396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512308    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.512396 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512288    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512499    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512506    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512540    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512542    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512553    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512553    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512573    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512575    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nvz\""
Oct 30 21:49:37.512589 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512585    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:49:37.512874 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512614    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:49:37.512874 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512619    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.512874 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512624    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.512874 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512645    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.512874 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512645    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.512874 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.512750    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:49:37.513116 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513097    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:49:37.513195 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513172    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:49:37.513251 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513240    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.513287 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513260    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.513468 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513455    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:49:37.513547 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513537    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c6mrn\""
Oct 30 21:49:37.513603 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513593    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:49:37.513661 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513651    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:49:37.513696 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513676    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:37.513723 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.513715    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-84vh4\""
Oct 30 21:49:37.518271 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59dafadb-fce7-4a75-afc2-fc4275882723-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:37.518347 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518284    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ssfj\" (UniqueName: \"kubernetes.io/projected/d07ee097-855f-4598-a947-2268077bac00-kube-api-access-6ssfj\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.518347 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518313    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.518421 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.518451 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518434    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.518481 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.518508 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-stats-auth\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.518535 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2q97\" (UniqueName: \"kubernetes.io/projected/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-kube-api-access-s2q97\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.518601 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518535    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.518601 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518565    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.518601 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4775f051-dc74-47a3-b528-a735daac1287-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.518701 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518615    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.518701 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-config\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.518701 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-trusted-ca\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.518835 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518717    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.518835 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxhv\" (UniqueName: \"kubernetes.io/projected/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-kube-api-access-wfxhv\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.518835 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518795    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8n9\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.518835 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518823    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-serving-cert\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.519010 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.519010 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518876    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-cert\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.519010 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472cs\" (UniqueName: \"kubernetes.io/projected/1fdc58d1-11b9-4879-8879-8909f186bbe5-kube-api-access-472cs\") pod \"kas-connection-checker-685895c7c9-jqx4n\" (UID: \"1fdc58d1-11b9-4879-8879-8909f186bbe5\") " pod="kube-system/kas-connection-checker-685895c7c9-jqx4n"
Oct 30 21:49:37.519010 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518942    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.519010 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.518982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.519224 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmmt\" (UniqueName: \"kubernetes.io/projected/bf193224-26cc-4cd6-8aa5-2424e1212c2a-kube-api-access-7cmmt\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.519224 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519071    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwm4\" (UniqueName: \"kubernetes.io/projected/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-kube-api-access-scwm4\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.519224 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.519224 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-serving-cert\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.519224 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-config\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.519224 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519193    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sc26\" (UniqueName: \"kubernetes.io/projected/4775f051-dc74-47a3-b528-a735daac1287-kube-api-access-6sc26\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.519387 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519249    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.519387 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.519387 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519325    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mrs4\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.519387 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr2j\" (UniqueName: \"kubernetes.io/projected/920770c6-fd18-41d7-95f6-a55c195361e6-kube-api-access-6cr2j\") pod \"kas-connection-checker-685895c7c9-m87n9\" (UID: \"920770c6-fd18-41d7-95f6-a55c195361e6\") " pod="kube-system/kas-connection-checker-685895c7c9-m87n9"
Oct 30 21:49:37.519387 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519359    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4qp\" (UniqueName: \"kubernetes.io/projected/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-kube-api-access-ns4qp\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.519522 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.519522 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519488    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4775f051-dc74-47a3-b528-a735daac1287-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.519522 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519506    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2198f51-8630-484c-9825-bb91b3b897e4-config-volume\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.519613 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519561    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.519660 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dssr\" (UniqueName: \"kubernetes.io/projected/47bc5387-3f28-4d19-9f51-cdd6e298896f-kube-api-access-6dssr\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.519660 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519634    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.519763 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519699    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plqkh\" (UniqueName: \"kubernetes.io/projected/f00a23f9-5c7e-437b-929c-4f793b200fca-kube-api-access-plqkh\") pod \"network-check-source-6f78d59f99-t9z6m\" (UID: \"f00a23f9-5c7e-437b-929c-4f793b200fca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"
Oct 30 21:49:37.519763 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519729    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.519763 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519757    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnmz\" (UniqueName: \"kubernetes.io/projected/9a549fa1-91e2-4c55-b306-2baf8e54dd96-kube-api-access-4fnmz\") pod \"volume-data-source-validator-58c9f5f46d-nxqdw\" (UID: \"9a549fa1-91e2-4c55-b306-2baf8e54dd96\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"
Oct 30 21:49:37.519868 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519817    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42fq\" (UniqueName: \"kubernetes.io/projected/c2198f51-8630-484c-9825-bb91b3b897e4-kube-api-access-m42fq\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.519868 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.519962 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07ee097-855f-4598-a947-2268077bac00-serving-cert\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.519962 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519918    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-snapshots\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.519962 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519942    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:37.520087 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.519982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.520087 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-default-certificate\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.520087 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf193224-26cc-4cd6-8aa5-2424e1212c2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.520202 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-tmp\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.520202 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520110    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.520202 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520126    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8js\" (UniqueName: \"kubernetes.io/projected/9aaad021-a27a-4ee3-97d7-71fbaa1aff87-kube-api-access-gb8js\") pod \"kas-connection-checker-685895c7c9-w9w7t\" (UID: \"9aaad021-a27a-4ee3-97d7-71fbaa1aff87\") " pod="kube-system/kas-connection-checker-685895c7c9-w9w7t"
Oct 30 21:49:37.520202 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.520194    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:37.520394 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.520276    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.020251125 +0000 UTC m=+32.103080963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:37.520394 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520294    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2198f51-8630-484c-9825-bb91b3b897e4-tmp-dir\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.520394 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520322    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.520789 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.520774    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf193224-26cc-4cd6-8aa5-2424e1212c2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.563144 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.563102    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:49:37.563299 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.563146    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:49:37.563519 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.563500    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:49:37.563920 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.563901    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"]
Oct 30 21:49:37.583514 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.583479    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr2j\" (UniqueName: \"kubernetes.io/projected/920770c6-fd18-41d7-95f6-a55c195361e6-kube-api-access-6cr2j\") pod \"kas-connection-checker-685895c7c9-m87n9\" (UID: \"920770c6-fd18-41d7-95f6-a55c195361e6\") " pod="kube-system/kas-connection-checker-685895c7c9-m87n9"
Oct 30 21:49:37.583626 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.583546    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-472cs\" (UniqueName: \"kubernetes.io/projected/1fdc58d1-11b9-4879-8879-8909f186bbe5-kube-api-access-472cs\") pod \"kas-connection-checker-685895c7c9-jqx4n\" (UID: \"1fdc58d1-11b9-4879-8879-8909f186bbe5\") " pod="kube-system/kas-connection-checker-685895c7c9-jqx4n"
Oct 30 21:49:37.583755 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.583736    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmmt\" (UniqueName: \"kubernetes.io/projected/bf193224-26cc-4cd6-8aa5-2424e1212c2a-kube-api-access-7cmmt\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:37.621629 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4qp\" (UniqueName: \"kubernetes.io/projected/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-kube-api-access-ns4qp\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.621785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.621785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4775f051-dc74-47a3-b528-a735daac1287-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.621785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621674    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2198f51-8630-484c-9825-bb91b3b897e4-config-volume\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.621785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.621785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621719    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dssr\" (UniqueName: \"kubernetes.io/projected/47bc5387-3f28-4d19-9f51-cdd6e298896f-kube-api-access-6dssr\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.621785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621739    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.621785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqkh\" (UniqueName: \"kubernetes.io/projected/f00a23f9-5c7e-437b-929c-4f793b200fca-kube-api-access-plqkh\") pod \"network-check-source-6f78d59f99-t9z6m\" (UID: \"f00a23f9-5c7e-437b-929c-4f793b200fca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621816    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnmz\" (UniqueName: \"kubernetes.io/projected/9a549fa1-91e2-4c55-b306-2baf8e54dd96-kube-api-access-4fnmz\") pod \"volume-data-source-validator-58c9f5f46d-nxqdw\" (UID: \"9a549fa1-91e2-4c55-b306-2baf8e54dd96\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.621835    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621848    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42fq\" (UniqueName: \"kubernetes.io/projected/c2198f51-8630-484c-9825-bb91b3b897e4-kube-api-access-m42fq\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.621899    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07ee097-855f-4598-a947-2268077bac00-serving-cert\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.621929    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.121909932 +0000 UTC m=+32.204739755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.621933    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.621949    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.621962    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.121943545 +0000 UTC m=+32.204773366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.622002    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.121988144 +0000 UTC m=+32.204817971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.621996    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-snapshots\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.622163 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-default-certificate\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622151    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-tmp\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8js\" (UniqueName: \"kubernetes.io/projected/9aaad021-a27a-4ee3-97d7-71fbaa1aff87-kube-api-access-gb8js\") pod \"kas-connection-checker-685895c7c9-w9w7t\" (UID: \"9aaad021-a27a-4ee3-97d7-71fbaa1aff87\") " pod="kube-system/kas-connection-checker-685895c7c9-w9w7t"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2198f51-8630-484c-9825-bb91b3b897e4-tmp-dir\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.622261    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622278    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59dafadb-fce7-4a75-afc2-fc4275882723-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.622333    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.122317416 +0000 UTC m=+32.205147233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622339    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2198f51-8630-484c-9825-bb91b3b897e4-config-volume\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622354    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssfj\" (UniqueName: \"kubernetes.io/projected/d07ee097-855f-4598-a947-2268077bac00-kube-api-access-6ssfj\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622421    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622510    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-stats-auth\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622537    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2q97\" (UniqueName: \"kubernetes.io/projected/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-kube-api-access-s2q97\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.622992 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622570    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622597    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4775f051-dc74-47a3-b528-a735daac1287-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.622642    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622650    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-config\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.622691    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.122678605 +0000 UTC m=+32.205508419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : secret "router-metrics-certs-default" not found
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-trusted-ca\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxhv\" (UniqueName: \"kubernetes.io/projected/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-kube-api-access-wfxhv\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622806    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8n9\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622820    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-snapshots\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622832    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-serving-cert\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622858    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622883    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-cert\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.623795 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622930    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-tmp\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622936    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.622993    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwm4\" (UniqueName: \"kubernetes.io/projected/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-kube-api-access-scwm4\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623014    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59dafadb-fce7-4a75-afc2-fc4275882723-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-serving-cert\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623081    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-config\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sc26\" (UniqueName: \"kubernetes.io/projected/4775f051-dc74-47a3-b528-a735daac1287-kube-api-access-6sc26\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mrs4\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.623311    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623341    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-config\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.623356    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.123340555 +0000 UTC m=+32.206170359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623369    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.624695 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.623548    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:37.625443 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:37.623594    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:38.12357863 +0000 UTC m=+32.206408446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:37.625443 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.623839    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.625443 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.624109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.625443 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.624356    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2198f51-8630-484c-9825-bb91b3b897e4-tmp-dir\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.625443 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.624690    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07ee097-855f-4598-a947-2268077bac00-serving-cert\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.625443 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.624827    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-config\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.625443 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.624949    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.625785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.625657    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4775f051-dc74-47a3-b528-a735daac1287-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.626079 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.626052    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-stats-auth\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.626079 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.626074    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.626241 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.626109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4775f051-dc74-47a3-b528-a735daac1287-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.626314 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.626250    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.626581 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.626557    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.626662 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.626597    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-default-certificate\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.627112 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.627068    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m87n9"
Oct 30 21:49:37.627617 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.627591    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-trusted-ca\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.628596 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.628569    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.628880 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.628852    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.628960 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.628928    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.629056 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.629032    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.629141 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.629115    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-serving-cert\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.629291 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.629272    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.629555 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.629532    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-serving-cert\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.629671 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.629628    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jqx4n"
Oct 30 21:49:37.630057 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.630040    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.630787 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.630771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-cert\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.685340 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.685278    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42fq\" (UniqueName: \"kubernetes.io/projected/c2198f51-8630-484c-9825-bb91b3b897e4-kube-api-access-m42fq\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:37.685498 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.685378    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4qp\" (UniqueName: \"kubernetes.io/projected/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-kube-api-access-ns4qp\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:37.685498 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.685448    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dssr\" (UniqueName: \"kubernetes.io/projected/47bc5387-3f28-4d19-9f51-cdd6e298896f-kube-api-access-6dssr\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:37.686195 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.686165    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqkh\" (UniqueName: \"kubernetes.io/projected/f00a23f9-5c7e-437b-929c-4f793b200fca-kube-api-access-plqkh\") pod \"network-check-source-6f78d59f99-t9z6m\" (UID: \"f00a23f9-5c7e-437b-929c-4f793b200fca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"
Oct 30 21:49:37.690382 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.687232    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2q97\" (UniqueName: \"kubernetes.io/projected/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-kube-api-access-s2q97\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.690382 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.688546    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.691824 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.691499    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8js\" (UniqueName: \"kubernetes.io/projected/9aaad021-a27a-4ee3-97d7-71fbaa1aff87-kube-api-access-gb8js\") pod \"kas-connection-checker-685895c7c9-w9w7t\" (UID: \"9aaad021-a27a-4ee3-97d7-71fbaa1aff87\") " pod="kube-system/kas-connection-checker-685895c7c9-w9w7t"
Oct 30 21:49:37.692018 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.691925    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnmz\" (UniqueName: \"kubernetes.io/projected/9a549fa1-91e2-4c55-b306-2baf8e54dd96-kube-api-access-4fnmz\") pod \"volume-data-source-validator-58c9f5f46d-nxqdw\" (UID: \"9a549fa1-91e2-4c55-b306-2baf8e54dd96\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"
Oct 30 21:49:37.694516 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.693755    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sc26\" (UniqueName: \"kubernetes.io/projected/4775f051-dc74-47a3-b528-a735daac1287-kube-api-access-6sc26\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.699346 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.698017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxhv\" (UniqueName: \"kubernetes.io/projected/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-kube-api-access-wfxhv\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.699346 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.698582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:37.699346 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.699109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssfj\" (UniqueName: \"kubernetes.io/projected/d07ee097-855f-4598-a947-2268077bac00-kube-api-access-6ssfj\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.703642 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.701564    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mrs4\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:37.704646 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.704549    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.705754 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.705729    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8n9\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:37.706582 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.706523    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwm4\" (UniqueName: \"kubernetes.io/projected/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-kube-api-access-scwm4\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.712482 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.712442    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"
Oct 30 21:49:37.737515 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.737486    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:37.750265 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.750246    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:37.757883 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.757855    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"
Oct 30 21:49:37.762779 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.762620    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:37.883199 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.883173    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:37.973262 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.973167    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:37.984445 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:37.984421    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w9w7t"
Oct 30 21:49:38.014383 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.014343    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jqx4n"]
Oct 30 21:49:38.027606 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.027573    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:38.027798 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.027781    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:38.027856 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.027839    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.02782423 +0000 UTC m=+33.110654030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:38.029450 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.028519    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m87n9"]
Oct 30 21:49:38.037555 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.037524    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920770c6_fd18_41d7_95f6_a55c195361e6.slice/crio-e1d19ac77f621f194d7f843f3f312fd667cdc82536a41baffa2499562be69330 WatchSource:0}: Error finding container e1d19ac77f621f194d7f843f3f312fd667cdc82536a41baffa2499562be69330: Status 404 returned error can't find the container with id e1d19ac77f621f194d7f843f3f312fd667cdc82536a41baffa2499562be69330
Oct 30 21:49:38.110519 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.110487    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xh5b2"]
Oct 30 21:49:38.112735 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.112689    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"]
Oct 30 21:49:38.113928 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.113887    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84861d0_c9a6_480b_82ed_5a0ce96dab7d.slice/crio-589b48d1061ddbd25326b757f7cf01fc11c79ffe22b7f9d3369b9cb9f0ad77f3 WatchSource:0}: Error finding container 589b48d1061ddbd25326b757f7cf01fc11c79ffe22b7f9d3369b9cb9f0ad77f3: Status 404 returned error can't find the container with id 589b48d1061ddbd25326b757f7cf01fc11c79ffe22b7f9d3369b9cb9f0ad77f3
Oct 30 21:49:38.120868 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.120840    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a549fa1_91e2_4c55_b306_2baf8e54dd96.slice/crio-2763a23ba894a416b6d0c233ca992372f718767485552d8505aaedeee1f8ef55 WatchSource:0}: Error finding container 2763a23ba894a416b6d0c233ca992372f718767485552d8505aaedeee1f8ef55: Status 404 returned error can't find the container with id 2763a23ba894a416b6d0c233ca992372f718767485552d8505aaedeee1f8ef55
Oct 30 21:49:38.125524 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.125503    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:38.128783 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.128763    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:38.128859 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.128795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:38.128909 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.128893    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:38.128968 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.128910    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:49:38.128968 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.128929    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.128919001 +0000 UTC m=+33.211748836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:38.128968 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.128930    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.128969    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.128951244 +0000 UTC m=+33.211781057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : secret "router-metrics-certs-default" not found
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.128981    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.128975645 +0000 UTC m=+33.211805503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.128994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.129017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.129036    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.129061    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129071    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:38.129077 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129081    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129112    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.129101295 +0000 UTC m=+33.211931096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129116    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129136    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.129130386 +0000 UTC m=+33.211960226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129157    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129160    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129177    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129203    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.129192455 +0000 UTC m=+33.212022257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:38.129367 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:38.129279    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:39.129270964 +0000 UTC m=+33.212100765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:38.158095 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.158072    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"]
Oct 30 21:49:38.159329 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.159271    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fs49h"]
Oct 30 21:49:38.161474 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.161452    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07ee097_855f_4598_a947_2268077bac00.slice/crio-b614bb832af3e09b23b359998aca23ab58c5a9ab7952ccb92b54bca9183113d4 WatchSource:0}: Error finding container b614bb832af3e09b23b359998aca23ab58c5a9ab7952ccb92b54bca9183113d4: Status 404 returned error can't find the container with id b614bb832af3e09b23b359998aca23ab58c5a9ab7952ccb92b54bca9183113d4
Oct 30 21:49:38.164487 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.164465    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c89d90_bbaa_4f37_acb6_c83e561a1d72.slice/crio-468bd96adaec12e1b28d6e1a218f3e03184c2a84f59b7c71df0cf693d0ed46af WatchSource:0}: Error finding container 468bd96adaec12e1b28d6e1a218f3e03184c2a84f59b7c71df0cf693d0ed46af: Status 404 returned error can't find the container with id 468bd96adaec12e1b28d6e1a218f3e03184c2a84f59b7c71df0cf693d0ed46af
Oct 30 21:49:38.178683 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.178665    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"]
Oct 30 21:49:38.181244 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.181202    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00a23f9_5c7e_437b_929c_4f793b200fca.slice/crio-a4e122d178826bc4ca4a231963dd65f5e698fa34878685473f5bd1f808c64d32 WatchSource:0}: Error finding container a4e122d178826bc4ca4a231963dd65f5e698fa34878685473f5bd1f808c64d32: Status 404 returned error can't find the container with id a4e122d178826bc4ca4a231963dd65f5e698fa34878685473f5bd1f808c64d32
Oct 30 21:49:38.182233 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.182194    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrgwp"]
Oct 30 21:49:38.184445 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.184426    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaec4c99_a4dd_47e4_9f3d_f6a276f166a0.slice/crio-e3cc2ece358f944d05196764702fea888b194d309b629fc34e79aa1706ea28d7 WatchSource:0}: Error finding container e3cc2ece358f944d05196764702fea888b194d309b629fc34e79aa1706ea28d7: Status 404 returned error can't find the container with id e3cc2ece358f944d05196764702fea888b194d309b629fc34e79aa1706ea28d7
Oct 30 21:49:38.349532 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.349509    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"]
Oct 30 21:49:38.352423 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.352401    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4775f051_dc74_47a3_b528_a735daac1287.slice/crio-f1ab68e70041e210fc7f8c303e709329e0fa1234babd891c1f6a74d78e8bde77 WatchSource:0}: Error finding container f1ab68e70041e210fc7f8c303e709329e0fa1234babd891c1f6a74d78e8bde77: Status 404 returned error can't find the container with id f1ab68e70041e210fc7f8c303e709329e0fa1234babd891c1f6a74d78e8bde77
Oct 30 21:49:38.358190 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.358172    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:38.358286 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.358201    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:38.365087 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.365059    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w9w7t"]
Oct 30 21:49:38.368785 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:38.368754    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aaad021_a27a_4ee3_97d7_71fbaa1aff87.slice/crio-6245e620c4fd603bb30da8e0539721bab255b15781717543fee7786d05b7e371 WatchSource:0}: Error finding container 6245e620c4fd603bb30da8e0539721bab255b15781717543fee7786d05b7e371: Status 404 returned error can't find the container with id 6245e620c4fd603bb30da8e0539721bab255b15781717543fee7786d05b7e371
Oct 30 21:49:38.415432 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.415408    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jjx9q\""
Oct 30 21:49:38.415432 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.415421    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:49:38.415432 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.415437    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gmxqg\""
Oct 30 21:49:38.554404 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.554362    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s" event={"ID":"a7c89d90-bbaa-4f37-acb6-c83e561a1d72","Type":"ContainerStarted","Data":"468bd96adaec12e1b28d6e1a218f3e03184c2a84f59b7c71df0cf693d0ed46af"}
Oct 30 21:49:38.555452 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.555431    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xh5b2" event={"ID":"e84861d0-c9a6-480b-82ed-5a0ce96dab7d","Type":"ContainerStarted","Data":"589b48d1061ddbd25326b757f7cf01fc11c79ffe22b7f9d3369b9cb9f0ad77f3"}
Oct 30 21:49:38.556443 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.556425    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m" event={"ID":"f00a23f9-5c7e-437b-929c-4f793b200fca","Type":"ContainerStarted","Data":"a4e122d178826bc4ca4a231963dd65f5e698fa34878685473f5bd1f808c64d32"}
Oct 30 21:49:38.557836 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.557819    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jqx4n" event={"ID":"1fdc58d1-11b9-4879-8879-8909f186bbe5","Type":"ContainerStarted","Data":"41bb82ad3412d51d59755325c21c9b8705a0cce1e4c5fdd886bc1a02717b749c"}
Oct 30 21:49:38.557902 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.557852    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jqx4n" event={"ID":"1fdc58d1-11b9-4879-8879-8909f186bbe5","Type":"ContainerStarted","Data":"3ea14ac24914466f3ff350290786b2fb621a3ec9b831e25d78f49b740ee590ba"}
Oct 30 21:49:38.559224 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.559187    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w9w7t" event={"ID":"9aaad021-a27a-4ee3-97d7-71fbaa1aff87","Type":"ContainerStarted","Data":"d44e14a56074e196d0742cb752c044817dc52bf062c47b3546e3280531c85aff"}
Oct 30 21:49:38.559298 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.559240    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w9w7t" event={"ID":"9aaad021-a27a-4ee3-97d7-71fbaa1aff87","Type":"ContainerStarted","Data":"6245e620c4fd603bb30da8e0539721bab255b15781717543fee7786d05b7e371"}
Oct 30 21:49:38.560302 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.560283    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r" event={"ID":"4775f051-dc74-47a3-b528-a735daac1287","Type":"ContainerStarted","Data":"f1ab68e70041e210fc7f8c303e709329e0fa1234babd891c1f6a74d78e8bde77"}
Oct 30 21:49:38.561323 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.561305    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrgwp" event={"ID":"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0","Type":"ContainerStarted","Data":"e3cc2ece358f944d05196764702fea888b194d309b629fc34e79aa1706ea28d7"}
Oct 30 21:49:38.562287 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.562267    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h" event={"ID":"d07ee097-855f-4598-a947-2268077bac00","Type":"ContainerStarted","Data":"b614bb832af3e09b23b359998aca23ab58c5a9ab7952ccb92b54bca9183113d4"}
Oct 30 21:49:38.563379 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.563362    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw" event={"ID":"9a549fa1-91e2-4c55-b306-2baf8e54dd96","Type":"ContainerStarted","Data":"2763a23ba894a416b6d0c233ca992372f718767485552d8505aaedeee1f8ef55"}
Oct 30 21:49:38.564631 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.564615    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m87n9" event={"ID":"920770c6-fd18-41d7-95f6-a55c195361e6","Type":"ContainerStarted","Data":"a7b3bb73eb78737fcb89031efe6da4984a183817d935a30f4992543975d49533"}
Oct 30 21:49:38.564712 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:38.564644    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m87n9" event={"ID":"920770c6-fd18-41d7-95f6-a55c195361e6","Type":"ContainerStarted","Data":"e1d19ac77f621f194d7f843f3f312fd667cdc82536a41baffa2499562be69330"}
Oct 30 21:49:39.039046 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.039002    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:39.039336 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.039228    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:39.039336 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.039297    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.039277774 +0000 UTC m=+35.122107591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.139835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.139880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.139945    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.139975    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.140016    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.140077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.140109    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140276    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140359    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.140343232 +0000 UTC m=+35.223173045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140555    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140609    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.140595784 +0000 UTC m=+35.223425601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140717    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140731    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140766    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.140754935 +0000 UTC m=+35.223584750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140824    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140836    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:39.141110 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140867    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.140857736 +0000 UTC m=+35.223687555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:39.142114 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140926    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:39.142114 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.140959    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.140949042 +0000 UTC m=+35.223778847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:39.142114 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.141024    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:49:39.142114 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.141081    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.14107035 +0000 UTC m=+35.223900153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : secret "router-metrics-certs-default" not found
Oct 30 21:49:39.142538 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:39.142510    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:41.142492029 +0000 UTC m=+35.225321844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:39.614318 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.612563    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jqx4n" podStartSLOduration=393.61254778 podStartE2EDuration="6m33.61254778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.612170738 +0000 UTC m=+33.695000561" watchObservedRunningTime="2026-10-30 21:49:39.61254778 +0000 UTC m=+33.695377602"
Oct 30 21:49:39.849454 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.847617    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w9w7t" podStartSLOduration=393.847598174 podStartE2EDuration="6m33.847598174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.730999522 +0000 UTC m=+33.813829346" watchObservedRunningTime="2026-10-30 21:49:39.847598174 +0000 UTC m=+33.930427998"
Oct 30 21:49:39.849454 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:39.848784    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-m87n9" podStartSLOduration=393.84877043 podStartE2EDuration="6m33.84877043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.846793295 +0000 UTC m=+33.929623115" watchObservedRunningTime="2026-10-30 21:49:39.84877043 +0000 UTC m=+33.931600254"
Oct 30 21:49:40.151321 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.150362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:40.151321 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.150448    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:40.151321 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:40.151005    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:49:40.151321 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:40.151069    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:12.15105109 +0000 UTC m=+66.233880910 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : secret "metrics-daemon-secret" not found
Oct 30 21:49:40.161585 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.161529    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:40.177693 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.177477    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:40.251954 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.251906    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:40.268981 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.268946    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:40.473947 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.473874    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:40.612322 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.611444    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw" event={"ID":"9a549fa1-91e2-4c55-b306-2baf8e54dd96","Type":"ContainerStarted","Data":"1f435485a3b01f7c85cfa227bfe70201ea1fe149fd58e0f9024442e350844b2d"}
Oct 30 21:49:40.643785 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.643726    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jl8q2"]
Oct 30 21:49:40.733578 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.732546    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw" podStartSLOduration=373.575849161 podStartE2EDuration="6m15.732528584s" totalImagesPullingTime="2.156679423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:40.731368502 +0000 UTC m=+34.814198325" watchObservedRunningTime="2026-10-30 21:49:40.732528584 +0000 UTC m=+34.815358408"
Oct 30 21:49:40.877743 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.877707    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjd7c"]
Oct 30 21:49:40.885658 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:40.885617    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a7ec18_5d7b_4dc1_89f1_58effdc9ba66.slice/crio-255554e0a0dbb590af06528ee7c1b20b7dc96950b2e39b104188f2f0d9de5482 WatchSource:0}: Error finding container 255554e0a0dbb590af06528ee7c1b20b7dc96950b2e39b104188f2f0d9de5482: Status 404 returned error can't find the container with id 255554e0a0dbb590af06528ee7c1b20b7dc96950b2e39b104188f2f0d9de5482
Oct 30 21:49:40.961514 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:40.961472    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:41.060389 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.060303    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:41.060671 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.060637    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:41.060802 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.060739    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.060701881 +0000 UTC m=+39.143531702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:41.161223 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.161158    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.161237    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.161277    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.161309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.161344    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.161408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.161440    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.161587    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:41.161656 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.161649    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.161631452 +0000 UTC m=+39.244461258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:41.162078 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162053    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.162035823 +0000 UTC m=+39.244865637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:41.162150 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162136    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:41.162194 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162150    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:41.162194 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162193    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.162182107 +0000 UTC m=+39.245011926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:41.162322 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162268    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:41.162322 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162278    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:41.162322 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162310    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.162298402 +0000 UTC m=+39.245128216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:41.162480 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162366    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:41.162480 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162395    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.162385549 +0000 UTC m=+39.245215354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:41.162480 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162442    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:41.162480 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162469    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.162460545 +0000 UTC m=+39.245290345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:41.162593 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162516    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:49:41.162593 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:41.162542    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.162535842 +0000 UTC m=+39.245365643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : secret "router-metrics-certs-default" not found
Oct 30 21:49:41.237365 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.237230    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:41.618435 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.618403    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjd7c" event={"ID":"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66","Type":"ContainerStarted","Data":"255554e0a0dbb590af06528ee7c1b20b7dc96950b2e39b104188f2f0d9de5482"}
Oct 30 21:49:41.622083 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:41.622052    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jl8q2" event={"ID":"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5","Type":"ContainerStarted","Data":"90b58fb6ab9014b5f6e8735e8734de09b20715a122397456737fe29b7e86a1d9"}
Oct 30 21:49:42.882369 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:42.882331    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:43.382788 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:43.382752    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:43.632017 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:43.631973    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xh5b2" event={"ID":"e84861d0-c9a6-480b-82ed-5a0ce96dab7d","Type":"ContainerStarted","Data":"44eb39b26537d74fd985f27f07600555405efe3898eaf919ab4393c71e62313e"}
Oct 30 21:49:43.635316 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:43.635290    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/0.log"
Oct 30 21:49:43.635478 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:43.635328    2572 generic.go:356] "Generic (PLEG): container finished" podID="d07ee097-855f-4598-a947-2268077bac00" containerID="d6d2c610ad50272e031d6437e461359f5c9fec56d52ac096375c24557965a686" exitCode=255
Oct 30 21:49:43.635478 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:43.635379    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h" event={"ID":"d07ee097-855f-4598-a947-2268077bac00","Type":"ContainerDied","Data":"d6d2c610ad50272e031d6437e461359f5c9fec56d52ac096375c24557965a686"}
Oct 30 21:49:43.635746 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:43.635629    2572 scope.go:122] "RemoveContainer" containerID="d6d2c610ad50272e031d6437e461359f5c9fec56d52ac096375c24557965a686"
Oct 30 21:49:43.873786 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:43.873736    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xh5b2" podStartSLOduration=375.860506517 podStartE2EDuration="6m20.873712115s" totalImagesPullingTime="5.013205598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:43.754103324 +0000 UTC m=+37.836933143" watchObservedRunningTime="2026-10-30 21:49:43.873712115 +0000 UTC m=+37.956541938"
Oct 30 21:49:44.583500 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.583422    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hvqh6"]
Oct 30 21:49:44.613710 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.613662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.641194 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.641153    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvqh6"]
Oct 30 21:49:44.650190 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.650158    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r" event={"ID":"4775f051-dc74-47a3-b528-a735daac1287","Type":"ContainerStarted","Data":"4a387337f1110c4df6ceec9c6fb732539ecdd51215bd4cb8428fc37267f92c21"}
Oct 30 21:49:44.667305 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.667268    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrgwp" event={"ID":"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0","Type":"ContainerStarted","Data":"afa771593afac4763cdd4df9fcab6c7b8a2a6c97768698ca77c422b9bddfa70f"}
Oct 30 21:49:44.668230 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.668184    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:49:44.668523 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.668503    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:49:44.668670 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.668621    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lwnsl\""
Oct 30 21:49:44.680136 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.680010    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:49:44.681607 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.680696    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/0.log"
Oct 30 21:49:44.681607 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.680734    2572 generic.go:356] "Generic (PLEG): container finished" podID="d07ee097-855f-4598-a947-2268077bac00" containerID="0124511bc3d9d40471df983f4e5b72596bc0e86b208962998ff187d6d35cf3be" exitCode=255
Oct 30 21:49:44.681607 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.680858    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h" event={"ID":"d07ee097-855f-4598-a947-2268077bac00","Type":"ContainerDied","Data":"0124511bc3d9d40471df983f4e5b72596bc0e86b208962998ff187d6d35cf3be"}
Oct 30 21:49:44.681607 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.680932    2572 scope.go:122] "RemoveContainer" containerID="d6d2c610ad50272e031d6437e461359f5c9fec56d52ac096375c24557965a686"
Oct 30 21:49:44.681607 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.681306    2572 scope.go:122] "RemoveContainer" containerID="0124511bc3d9d40471df983f4e5b72596bc0e86b208962998ff187d6d35cf3be"
Oct 30 21:49:44.681607 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:44.681509    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fs49h_openshift-console-operator(d07ee097-855f-4598-a947-2268077bac00)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h" podUID="d07ee097-855f-4598-a947-2268077bac00"
Oct 30 21:49:44.684854 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.684829    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s" event={"ID":"a7c89d90-bbaa-4f37-acb6-c83e561a1d72","Type":"ContainerStarted","Data":"21aa16c98a1df39f01c26c0231013e5ac3be15691d0d87183c3bcf5b9bc32117"}
Oct 30 21:49:44.717578 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.717542    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8aed736-cadf-4091-b78e-1d555860a055-data-volume\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.717752 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.717628    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbl8q\" (UniqueName: \"kubernetes.io/projected/d8aed736-cadf-4091-b78e-1d555860a055-kube-api-access-mbl8q\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.717816 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.717747    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.717870 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.717812    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.717920 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.717872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8aed736-cadf-4091-b78e-1d555860a055-crio-socket\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.789810 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.789774    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-scxcw-zq8dc"]
Oct 30 21:49:44.814007 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.813981    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:44.819155 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.819128    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8aed736-cadf-4091-b78e-1d555860a055-data-volume\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.819331 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.819316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbl8q\" (UniqueName: \"kubernetes.io/projected/d8aed736-cadf-4091-b78e-1d555860a055-kube-api-access-mbl8q\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.819504 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.819476    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8aed736-cadf-4091-b78e-1d555860a055-data-volume\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.819578 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.819479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.819633 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.819607    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.819698 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.819667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8aed736-cadf-4091-b78e-1d555860a055-crio-socket\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.821104 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:44.820657    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:49:44.821104 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:44.820720    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.320698585 +0000 UTC m=+39.403528573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : configmap "kube-rbac-proxy" not found
Oct 30 21:49:44.821104 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:44.820844    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:44.821104 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:44.820907    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.320883414 +0000 UTC m=+39.403713229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:44.821104 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.820915    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8aed736-cadf-4091-b78e-1d555860a055-crio-socket\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.852462 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.852384    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-scxcw-zq8dc"]
Oct 30 21:49:44.855731 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.855605    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s" podStartSLOduration=381.984529356 podStartE2EDuration="6m27.855593104s" totalImagesPullingTime="5.871063748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:44.854059502 +0000 UTC m=+38.936889323" watchObservedRunningTime="2026-10-30 21:49:44.855593104 +0000 UTC m=+38.938422926"
Oct 30 21:49:44.858959 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.858934    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:44.867646 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.867624    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qmkfs\""
Oct 30 21:49:44.884766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.884741    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbl8q\" (UniqueName: \"kubernetes.io/projected/d8aed736-cadf-4091-b78e-1d555860a055-kube-api-access-mbl8q\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:44.920437 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.920395    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:44.920635 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.920589    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:44.920811 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.920657    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:44.920811 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.920683    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7q47\" (UniqueName: \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:44.976500 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:44.976442    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xrgwp" podStartSLOduration=1.83146028 podStartE2EDuration="7.976428835s" totalImagesPullingTime="6.144968555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:44.975613693 +0000 UTC m=+39.058443515" watchObservedRunningTime="2026-10-30 21:49:44.976428835 +0000 UTC m=+39.059258658"
Oct 30 21:49:45.021421 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.021389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.021584 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.021497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.021584 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.021538    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.021584 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.021558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7q47\" (UniqueName: \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.021842 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.021815    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.022006 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.021985    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.022436 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.022414    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.076287 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.076252    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:45.084540 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.084513    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7q47\" (UniqueName: \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.097222 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.097162    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r" podStartSLOduration=378.396520492 podStartE2EDuration="6m24.097150272s" totalImagesPullingTime="5.70062978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:45.095818201 +0000 UTC m=+39.178648025" watchObservedRunningTime="2026-10-30 21:49:45.097150272 +0000 UTC m=+39.179980093"
Oct 30 21:49:45.123258 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.122886    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:45.124020 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.123995    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:45.124129 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.124073    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.124053636 +0000 UTC m=+47.206883457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:45.136465 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.136439    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:45.224354 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.224320    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.224368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.224400    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224497    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.224474913 +0000 UTC m=+47.307304731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224506    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224525    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224535    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224549    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.224531    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224586    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.224570075 +0000 UTC m=+47.307399890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224595    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:45.224601 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224612    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.224598877 +0000 UTC m=+47.307428682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224642    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.224632705 +0000 UTC m=+47.307462516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.224670    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224737    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.224760    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224770    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.22475953 +0000 UTC m=+47.307589348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.224807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224813    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224892    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224925    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.224917186 +0000 UTC m=+47.307746987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : secret "router-metrics-certs-default" not found
Oct 30 21:49:45.225073 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.224942    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.224930206 +0000 UTC m=+47.307760028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:45.325916 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.325889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:45.326021 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.325959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:45.326083 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.326052    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:45.326136 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.326098    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:49:45.326136 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.326126    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:46.326105409 +0000 UTC m=+40.408935232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:45.326263 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.326157    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:46.326140077 +0000 UTC m=+40.408969879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : configmap "kube-rbac-proxy" not found
Oct 30 21:49:45.667081 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.667000    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:45.690178 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.690142    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-scxcw-zq8dc"]
Oct 30 21:49:45.691755 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.691738    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:49:45.692707 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.692132    2572 scope.go:122] "RemoveContainer" containerID="0124511bc3d9d40471df983f4e5b72596bc0e86b208962998ff187d6d35cf3be"
Oct 30 21:49:45.692707 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:45.692389    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fs49h_openshift-console-operator(d07ee097-855f-4598-a947-2268077bac00)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h" podUID="d07ee097-855f-4598-a947-2268077bac00"
Oct 30 21:49:45.693632 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.693607    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjd7c" event={"ID":"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66","Type":"ContainerStarted","Data":"3f0f63c1162f14f0f0341f5494c597303d875d08f7b060da892403128ed5e840"}
Oct 30 21:49:45.693741 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.693707    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:45.694682 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:45.694659    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62848175_7827_47e2_af47_7edd2af96935.slice/crio-00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3 WatchSource:0}: Error finding container 00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3: Status 404 returned error can't find the container with id 00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3
Oct 30 21:49:45.695172 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.695145    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m" event={"ID":"f00a23f9-5c7e-437b-929c-4f793b200fca","Type":"ContainerStarted","Data":"a88d9238fdd284ab6fca59c72d7e2bffab2566fcddd5665d71db11c9d121a688"}
Oct 30 21:49:45.921640 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.921557    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:45.951925 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:45.951868    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m" podStartSLOduration=362.806590686 podStartE2EDuration="6m9.951849217s" totalImagesPullingTime="7.145258531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:45.951582013 +0000 UTC m=+40.034411841" watchObservedRunningTime="2026-10-30 21:49:45.951849217 +0000 UTC m=+40.034679041"
Oct 30 21:49:46.032064 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.032030    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"]
Oct 30 21:49:46.049700 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.049669    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"]
Oct 30 21:49:46.049880 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.049787    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"
Oct 30 21:49:46.067876 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.067855    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"
Oct 30 21:49:46.074312 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.074257    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cjd7c" podStartSLOduration=35.647501804 podStartE2EDuration="40.074244869s" totalImagesPullingTime="4.426743065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:46.073574006 +0000 UTC m=+40.156403830" watchObservedRunningTime="2026-10-30 21:49:46.074244869 +0000 UTC m=+40.157074690"
Oct 30 21:49:46.095660 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.095630    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"]
Oct 30 21:49:46.104333 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.104295    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bhhxg\""
Oct 30 21:49:46.104605 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.104585    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:46.104728 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.104703    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:49:46.110331 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.110308    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"]
Oct 30 21:49:46.134277 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.134250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htbb\" (UniqueName: \"kubernetes.io/projected/30369506-d839-4e0b-a652-a3875a88b1b4-kube-api-access-9htbb\") pod \"migrator-66df5d496b-66hdp\" (UID: \"30369506-d839-4e0b-a652-a3875a88b1b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"
Oct 30 21:49:46.134475 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.134452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzc2\" (UniqueName: \"kubernetes.io/projected/5719f87e-6553-4207-8b40-766d750d8b4c-kube-api-access-jlzc2\") pod \"migrator-66df5d496b-k6mjh\" (UID: \"5719f87e-6553-4207-8b40-766d750d8b4c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"
Oct 30 21:49:46.189011 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.188933    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:46.235518 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.235172    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzc2\" (UniqueName: \"kubernetes.io/projected/5719f87e-6553-4207-8b40-766d750d8b4c-kube-api-access-jlzc2\") pod \"migrator-66df5d496b-k6mjh\" (UID: \"5719f87e-6553-4207-8b40-766d750d8b4c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"
Oct 30 21:49:46.235518 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.235307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htbb\" (UniqueName: \"kubernetes.io/projected/30369506-d839-4e0b-a652-a3875a88b1b4-kube-api-access-9htbb\") pod \"migrator-66df5d496b-66hdp\" (UID: \"30369506-d839-4e0b-a652-a3875a88b1b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"
Oct 30 21:49:46.298251 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.298198    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htbb\" (UniqueName: \"kubernetes.io/projected/30369506-d839-4e0b-a652-a3875a88b1b4-kube-api-access-9htbb\") pod \"migrator-66df5d496b-66hdp\" (UID: \"30369506-d839-4e0b-a652-a3875a88b1b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"
Oct 30 21:49:46.298387 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.298279    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzc2\" (UniqueName: \"kubernetes.io/projected/5719f87e-6553-4207-8b40-766d750d8b4c-kube-api-access-jlzc2\") pod \"migrator-66df5d496b-k6mjh\" (UID: \"5719f87e-6553-4207-8b40-766d750d8b4c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"
Oct 30 21:49:46.336565 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.336526    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:46.336745 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.336611    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:46.336745 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:46.336705    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:46.336846 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:46.336773    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:48.336758463 +0000 UTC m=+42.419588279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:46.336846 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:46.336779    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:49:46.336846 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:46.336833    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:48.336814761 +0000 UTC m=+42.419644568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : configmap "kube-rbac-proxy" not found
Oct 30 21:49:46.375661 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.375634    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"
Oct 30 21:49:46.400513 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.400310    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"
Oct 30 21:49:46.427993 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.427880    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:46.658520 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.658492    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:46.700226 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.700178    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-scxcw-zq8dc" event={"ID":"62848175-7827-47e2-af47-7edd2af96935","Type":"ContainerStarted","Data":"172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c"}
Oct 30 21:49:46.700680 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.700258    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-scxcw-zq8dc" event={"ID":"62848175-7827-47e2-af47-7edd2af96935","Type":"ContainerStarted","Data":"00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3"}
Oct 30 21:49:46.781020 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.780994    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"]
Oct 30 21:49:46.783430 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:46.783408    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5719f87e_6553_4207_8b40_766d750d8b4c.slice/crio-615d5a302f84cae2265ede00172cb57c26bdccd29fac16ef39be19422eb05cfc WatchSource:0}: Error finding container 615d5a302f84cae2265ede00172cb57c26bdccd29fac16ef39be19422eb05cfc: Status 404 returned error can't find the container with id 615d5a302f84cae2265ede00172cb57c26bdccd29fac16ef39be19422eb05cfc
Oct 30 21:49:46.819998 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.819935    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-scxcw-zq8dc" podStartSLOduration=2.819919835 podStartE2EDuration="2.819919835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:46.81861267 +0000 UTC m=+40.901442505" watchObservedRunningTime="2026-10-30 21:49:46.819919835 +0000 UTC m=+40.902749658"
Oct 30 21:49:46.828553 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:46.828485    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"]
Oct 30 21:49:46.831405 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:46.831374    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30369506_d839_4e0b_a652_a3875a88b1b4.slice/crio-a75c718e93f64cf58132066f8cef04845af0ee72b9270a8a2bf6d00aa18d3adb WatchSource:0}: Error finding container a75c718e93f64cf58132066f8cef04845af0ee72b9270a8a2bf6d00aa18d3adb: Status 404 returned error can't find the container with id a75c718e93f64cf58132066f8cef04845af0ee72b9270a8a2bf6d00aa18d3adb
Oct 30 21:49:47.058446 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.058366    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sbhs4"]
Oct 30 21:49:47.092515 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.092483    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.121170 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.121143    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sbhs4"]
Oct 30 21:49:47.147968 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.147932    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:49:47.148186 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.148167    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-td54n\""
Oct 30 21:49:47.152614 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.152592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkdxm\" (UniqueName: \"kubernetes.io/projected/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-kube-api-access-nkdxm\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.152728 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.152636    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-key\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.152728 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.152662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-cabundle\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.208614 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.208586    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:49:47.208614 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.208604    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:49:47.208850 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.208833    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:49:47.254018 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.253979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdxm\" (UniqueName: \"kubernetes.io/projected/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-kube-api-access-nkdxm\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.254191 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.254030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-key\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.254191 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.254065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-cabundle\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.254766 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.254721    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-cabundle\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.256791 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.256766    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-key\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.319749 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.319682    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdxm\" (UniqueName: \"kubernetes.io/projected/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-kube-api-access-nkdxm\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.413131 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.413095    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:47.444962 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.444901    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:47.705059 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.705024    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp" event={"ID":"30369506-d839-4e0b-a652-a3875a88b1b4","Type":"ContainerStarted","Data":"a75c718e93f64cf58132066f8cef04845af0ee72b9270a8a2bf6d00aa18d3adb"}
Oct 30 21:49:47.706367 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.706335    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh" event={"ID":"5719f87e-6553-4207-8b40-766d750d8b4c","Type":"ContainerStarted","Data":"615d5a302f84cae2265ede00172cb57c26bdccd29fac16ef39be19422eb05cfc"}
Oct 30 21:49:47.716468 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.716439    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:47.738374 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.738349    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:47.738467 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.738400    2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:47.738801 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.738787    2572 scope.go:122] "RemoveContainer" containerID="0124511bc3d9d40471df983f4e5b72596bc0e86b208962998ff187d6d35cf3be"
Oct 30 21:49:47.739038 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:47.739007    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fs49h_openshift-console-operator(d07ee097-855f-4598-a947-2268077bac00)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h" podUID="d07ee097-855f-4598-a947-2268077bac00"
Oct 30 21:49:47.821352 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:47.821293    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sbhs4"]
Oct 30 21:49:47.823741 ip-10-0-141-147 kubenswrapper[2572]: W1030 21:49:47.823700    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd08e0e_c491_487a_a4f6_a0955b41c4a0.slice/crio-1ed92975ef9d76ec301f851358b9aa6481d0fcfe8c10ff696833a843d32f5ed5 WatchSource:0}: Error finding container 1ed92975ef9d76ec301f851358b9aa6481d0fcfe8c10ff696833a843d32f5ed5: Status 404 returned error can't find the container with id 1ed92975ef9d76ec301f851358b9aa6481d0fcfe8c10ff696833a843d32f5ed5
Oct 30 21:49:48.127005 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:48.126899    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:48.365055 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:48.364316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:48.365055 ip-10-0-141-147 kubenswrapper[2572]: I1030 21:49:48.364395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:48.365055 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:48.364551    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:49:48.365055 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:48.364609    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:52.364591986 +0000 UTC m=+46.447421800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : configmap "kube-rbac-proxy" not found
Oct 30 21:49:48.365055 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:48.364977    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:48.365055 ip-10-0-141-147 kubenswrapper[2572]: E1030 21:49:48.365027    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:52.365012364 +0000 UTC m=+46.447842179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:48.408231 ip-10-0-141-147 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:49:48.418930 ip-10-0-141-147 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:49:48.419123 ip-10-0-141-147 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:49:48.419179 ip-10-0-141-147 systemd[1]: kubelet.service: Consumed 3.148s CPU time.
Oct 30 21:49:48.429711 ip-10-0-141-147 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:49:48.523602 ip-10-0-141-147 kubenswrapper[5031]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:48.523602 ip-10-0-141-147 kubenswrapper[5031]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:49:48.523602 ip-10-0-141-147 kubenswrapper[5031]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:48.523602 ip-10-0-141-147 kubenswrapper[5031]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:48.525833 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525739    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.525833 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525831    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.525833 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525836    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525841    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525847    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525851    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525854    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525857    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525859    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525862    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525864    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525867    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525870    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525872    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525875    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525877    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525879    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525882    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525884    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525887    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525889    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525891    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.525965 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525894    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525897    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525902    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525906    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525910    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525922    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525925    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525928    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525931    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525934    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525936    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525940    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525944    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525947    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525950    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525953    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525955    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525958    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525961    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.526534 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525963    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525965    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525968    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525970    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525973    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525976    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525978    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525981    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525983    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525987    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525992    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.525996    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526002    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526006    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526010    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526014    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526018    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526023    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526027    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526031    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.527036 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526037    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526041    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526046    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526050    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526055    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526059    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526063    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526070    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526076    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526081    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526086    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526091    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526095    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526100    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526104    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526107    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526110    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526114    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526117    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.527582 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526119    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526122    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526124    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526127    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526129    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526132    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526134    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526137    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526140    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526142    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526145    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526147    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526150    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526153    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526155    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526157    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526160    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526163    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526166    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526169    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.528298 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526171    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526174    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526176    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526182    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526186    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.526189    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527094    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527102    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527105    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527109    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527111    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527114    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527117    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527119    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527122    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527127    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527131    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527136    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527143    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.528982 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527150    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527156    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527161    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527166    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527172    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527176    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527181    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527184    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527189    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527193    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527197    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527200    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527223    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527228    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527232    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527236    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527240    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527242    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527245    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527247    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.529626 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527249    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527252    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527255    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527257    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527261    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527266    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527269    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527272    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527275    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527277    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527280    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527282    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527284    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527287    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527289    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527292    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527294    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527297    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527299    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527302    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.530259 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527305    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527307    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527309    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527312    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527315    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527317    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527320    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527323    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527325    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527329    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527333    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527337    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527342    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527346    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527350    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527354    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527356    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527359    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527362    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.530877 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527369    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527372    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527375    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527378    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527380    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527383    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527386    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527388    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527390    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527393    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527395    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527398    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527401    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527405    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527410    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527414    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527418    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527421    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527423    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527426    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.531574 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527428    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527431    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527433    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527435    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527438    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527440    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527442    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527445    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527447    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527450    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527452    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527454    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527462    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.527466    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528274    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528285    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528291    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528295    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528301    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.532152 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528305    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528308    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528311    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528314    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528317    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528320    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528324    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528328    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528332    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528336    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528341    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528344    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528346    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528349    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528352    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528356    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528361    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528365    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528369    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528372    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.532772 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528374    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528377    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528379    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528383    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528387    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528396    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528398    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528401    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528404    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528406    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528409    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528412    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528414    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528417    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528420    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528423    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528427    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528429    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528433    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.533440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528437    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528441    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528446    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528450    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528455    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528457    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528460    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528462    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528465    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528468    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528471    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528473    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528476    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528478    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528481    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528483    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528486    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528488    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528493    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.533999 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528496    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528499    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528502    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528505    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528507    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528509    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528512    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528515    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528518    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528522    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528527    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528531    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528535    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528538    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528541    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528543    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528546    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528548    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528551    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528553    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.534583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528556    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528558    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528561    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528563    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528566    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528568    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528571    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528573    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528576    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528578    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528580    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528586    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528588    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528592    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528594    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528599    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528603    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528608    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528612    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528616    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.535161 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528620    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528623    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.528626    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528745    5031 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528761    5031 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528768    5031 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528775    5031 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528782    5031 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528787    5031 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528795    5031 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528802    5031 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528807    5031 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528812    5031 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528816    5031 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528819    5031 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528822    5031 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528825    5031 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528841    5031 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528844    5031 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528847    5031 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528850    5031 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528853    5031 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528857    5031 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:49:48.535764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528862    5031 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528866    5031 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528869    5031 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528875    5031 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528880    5031 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528885    5031 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528891    5031 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528896    5031 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528901    5031 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528906    5031 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528911    5031 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528916    5031 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528921    5031 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528924    5031 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528927    5031 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528930    5031 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528933    5031 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528936    5031 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528940    5031 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528944    5031 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528947    5031 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528950    5031 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528952    5031 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528956    5031 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:49:48.536465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528960    5031 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528963    5031 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528966    5031 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528968    5031 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528971    5031 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528974    5031 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528977    5031 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528979    5031 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528985    5031 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528988    5031 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.528995    5031 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529000    5031 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529007    5031 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529013    5031 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529018    5031 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529023    5031 flags.go:64] FLAG: --help="false"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529028    5031 flags.go:64] FLAG: --hostname-override="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529033    5031 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529035    5031 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529038    5031 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529043    5031 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529049    5031 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529053    5031 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:49:48.537292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529057    5031 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529061    5031 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529064    5031 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529067    5031 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529070    5031 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529073    5031 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529076    5031 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529079    5031 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529082    5031 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529084    5031 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529089    5031 flags.go:64] FLAG: --lock-file=""
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529094    5031 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529098    5031 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529103    5031 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529117    5031 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529122    5031 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529127    5031 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529134    5031 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529138    5031 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529141    5031 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529144    5031 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529147    5031 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529152    5031 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529155    5031 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529161    5031 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:49:48.538142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529166    5031 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529170    5031 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529175    5031 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529180    5031 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529185    5031 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529189    5031 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529195    5031 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529220    5031 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529226    5031 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529230    5031 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529236    5031 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529241    5031 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529246    5031 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529249    5031 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529252    5031 flags.go:64] FLAG: --port="10250"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529255    5031 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529258    5031 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fe041f1d29ca0b79"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529263    5031 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529268    5031 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529273    5031 flags.go:64] FLAG: --register-node="true"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529277    5031 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529282    5031 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529287    5031 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529291    5031 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529297    5031 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529302    5031 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:49:48.539080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529307    5031 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529311    5031 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529314    5031 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529316    5031 flags.go:64] FLAG: --runonce="false"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529320    5031 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529323    5031 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529326    5031 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529329    5031 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529331    5031 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529335    5031 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529338    5031 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529341    5031 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529344    5031 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529347    5031 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529350    5031 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529352    5031 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529355    5031 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529358    5031 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529361    5031 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529366    5031 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529368    5031 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529373    5031 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529379    5031 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529384    5031 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529388    5031 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:49:48.540075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529392    5031 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529396    5031 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529400    5031 flags.go:64] FLAG: --v="2"
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529407    5031 flags.go:64] FLAG: --version="false"
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529413    5031 flags.go:64] FLAG: --vmodule=""
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529419    5031 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.529422    5031 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529589    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529595    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529601    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529606    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529609    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529614    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529619    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529623    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529628    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529633    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529637    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529642    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529645    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529648    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.540819 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529650    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529653    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529655    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529658    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529660    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529663    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529667    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529671    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529675    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529682    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529689    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529694    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529699    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529703    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529707    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529709    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529712    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529715    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529717    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.541478 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529720    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529722    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529725    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529727    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529729    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529733    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529738    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529741    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529746    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529750    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529754    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529758    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529763    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529768    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529772    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529777    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529781    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529785    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529789    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529792    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.542184 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529794    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529797    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529800    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529802    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529804    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529807    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529809    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529812    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529814    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529816    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529819    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529821    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529824    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529826    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529828    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529831    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529835    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529839    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529846    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529849    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.542869 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529853    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529857    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529860    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529864    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529868    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529871    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529875    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529879    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529883    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529887    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529892    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529896    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529899    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529904    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529908    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529911    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529915    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529919    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529923    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529926    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.543483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529930    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529934    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529937    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529941    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529958    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529964    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529969    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529974    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529979    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529983    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529989    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529994    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.529998    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.530048    5031 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530090    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530094    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.544092 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530097    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530100    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530103    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530106    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530110    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530115    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530119    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530123    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530127    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530132    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530136    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530141    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530145    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530150    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530153    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530156    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530158    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530161    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530163    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530165    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.544644 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530168    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530170    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530173    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530175    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530177    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530180    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530184    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530186    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530189    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530192    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530195    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530198    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530200    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530203    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530219    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530223    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530227    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530230    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530235    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530238    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.545427 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530242    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530245    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530248    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530250    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530252    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530258    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530262    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530266    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530269    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530273    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530276    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530281    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530285    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530288    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530292    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530294    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530297    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530299    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530303    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.546023 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530305    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530308    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530311    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530314    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530316    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530319    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530321    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530324    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530326    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530329    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530331    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530334    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530336    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530339    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530341    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530343    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530346    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530355    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530363    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.546701 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530367    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530371    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530375    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530380    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530385    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530390    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530394    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530398    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530403    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530407    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530411    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530416    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530422    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530426    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530430    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530435    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530440    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530443    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530447    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530449    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.547199 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530453    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530455    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530458    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530460    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530463    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.530465    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.534860    5031 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.534877    5031 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.534895    5031 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.534899    5031 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534966    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534974    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534979    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534984    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534988    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534991    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534993    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534996    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.534998    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.547839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535000    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535003    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535006    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535008    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535012    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535014    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535017    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535019    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535022    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535026    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535029    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535034    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535039    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535043    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535047    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535050    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535053    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535055    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535058    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535060    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.548409 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535064    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535067    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535070    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535072    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535075    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535077    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535080    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535082    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535085    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535087    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535089    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535092    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535094    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535097    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535099    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535103    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535107    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535111    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535114    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535118    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.549001 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535122    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535126    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535130    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535134    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535139    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535143    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535146    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535149    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535151    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535154    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535156    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535158    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535161    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535165    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535168    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535170    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535173    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535175    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535178    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.549600 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535181    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535183    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535186    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535188    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535190    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535193    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535195    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535198    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535200    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535203    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535224    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535228    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535232    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535239    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535245    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535250    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535255    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535260    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535263    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.550254 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535266    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535268    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535270    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535273    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535276    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535278    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535281    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535283    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535287    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535289    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535293    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535297    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535300    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535304    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535309    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535314    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535318    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535321    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.550825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535324    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.535360    5031 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535517    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535524    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535529    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535534    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535537    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535539    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535542    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535544    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535547    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535550    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535552    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535555    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535557    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535560    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.551360 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535562    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535564    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535567    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535569    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535572    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535574    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535577    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535580    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535582    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535585    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535587    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535590    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535592    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535596    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535599    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535603    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535607    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535609    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535613    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.551853 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535617    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535620    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535622    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535626    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535630    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535634    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535638    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535641    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535645    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535649    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535652    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535655    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535657    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535660    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535662    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535665    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535667    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535670    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535672    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.552616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535682    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535685    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535688    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535690    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535693    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535695    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535698    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535701    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535706    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535710    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535714    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535718    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535721    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535726    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535730    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535735    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535739    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535743    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535747    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.553223 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535751    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535755    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535759    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535763    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535768    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535772    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535777    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535799    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535803    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535807    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535811    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535815    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535819    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535823    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535827    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535830    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535833    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535836    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535839    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535841    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.553854 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535843    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535846    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535851    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535854    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535856    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535859    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535862    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535866    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535869    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535873    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535878    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535882    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535886    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535890    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535895    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.535928    5031 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:48.554655 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535963    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535966    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535969    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535974    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535978    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535981    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535984    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535988    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535992    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.535997    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536001    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536006    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536011    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536017    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536020    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536023    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536025    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536028    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536031    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:48.555166 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536035    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536037    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536040    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536043    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536045    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536048    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536051    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536055    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536058    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536062    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536066    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536071    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536075    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536079    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536083    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536087    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536091    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536095    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536099    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:48.555825 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536103    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536106    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536108    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536111    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536114    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536116    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536119    5031 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536121    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536124    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536127    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536130    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536133    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536135    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536139    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536142    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536144    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536147    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536149    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536152    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536154    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:48.556483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536157    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536160    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536162    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536166    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536170    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536175    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536180    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536184    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536189    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536194    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536199    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536203    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536223    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536227    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536231    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536235    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536239    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536243    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536246    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536250    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:48.557125 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536254    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536259    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536262    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536264    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536266    5031 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536271    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536273    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536276    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536278    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536281    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536283    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536286    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536288    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536291    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536294    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536296    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536298    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536301    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536303    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536306    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:48.557839 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536308    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536311    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536313    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536316    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536318    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536322    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536324    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:48.536327    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.536398    5031 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.537502    5031 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.537900    5031 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.538233    5031 server.go:1014] "Starting client certificate rotation"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.538473    5031 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.538704    5031 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:06 +0000 UTC" deadline="2028-07-09 23:48:51.508394846 +0000 UTC"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.538810    5031 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14833h59m2.969588582s"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.539647    5031 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:48.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.541095    5031 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:48.559167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.542354    5031 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:49:48.559167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.544189    5031 log.go:25] "Validated CRI v1 image API"
Oct 30 21:49:48.559167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.545267    5031 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:49:48.559167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.547496    5031 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8c007e03-792c-439e-9e15-c57cad5c676c:/dev/nvme0n1p4 da8bc8a4-ffc0-4f30-a246-8cd06ab03bff:/dev/nvme0n1p3]
Oct 30 21:49:48.559372 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.547517    5031 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ed92975ef9d76ec301f851358b9aa6481d0fcfe8c10ff696833a843d32f5ed5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ed92975ef9d76ec301f851358b9aa6481d0fcfe8c10ff696833a843d32f5ed5/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/255554e0a0dbb590af06528ee7c1b20b7dc96950b2e39b104188f2f0d9de5482/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/255554e0a0dbb590af06528ee7c1b20b7dc96950b2e39b104188f2f0d9de5482/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2587c6a50a557f5c522cb7d8c1589baba2d4e84bee219796d490ee6b2c4b23bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2587c6a50a557f5c522cb7d8c1589baba2d4e84bee219796d490ee6b2c4b23bd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2763a23ba894a416b6d0c233ca992372f718767485552d8505aaedeee1f8ef55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2763a23ba894a416b6d0c233ca992372f718767485552d8505aaedeee1f8ef55/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bfdbbadcc4984a12e470900f67e8c5b696bbd60975f278a5a16876aa76d3217/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bfdbbadcc4984a12e470900f67e8c5b696bbd60975f278a5a16876aa76d3217/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ea14ac24914466f3ff350290786b2fb621a3ec9b831e25d78f49b740ee590ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ea14ac24914466f3ff350290786b2fb621a3ec9b831e25d78f49b740ee590ba/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/468bd96adaec12e1b28d6e1a218f3e03184c2a84f59b7c71df0cf693d0ed46af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/468bd96adaec12e1b28d6e1a218f3e03184c2a84f59b7c71df0cf693d0ed46af/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4aa635da86fa6866138ac0a2957518a7984ddd913ef3e1c5fe487c13ef15ccf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4aa635da86fa6866138ac0a2957518a7984ddd913ef3e1c5fe487c13ef15ccf8/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dcc139cffd7b6c9b5f1e3d558d85a519322c05cde86d44a84cdbb44e2fa21e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dcc139cffd7b6c9b5f1e3d558d85a519322c05cde86d44a84cdbb44e2fa21e4/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55e1f0b2c6b61cd08c854054a7562c94b1d1a8a8b042ac432a7e876805dcb5c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55e1f0b2c6b61cd08c854054a7562c94b1d1a8a8b042ac432a7e876805dcb5c2/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/589b48d1061ddbd25326b757f7cf01fc11c79ffe22b7f9d3369b9cb9f0ad77f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/589b48d1061ddbd25326b757f7cf01fc11c79ffe22b7f9d3369b9cb9f0ad77f3/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ebb8f08c89cc3e2a841eef810f0ecfbff47a7d4487cb6075cac2b569fb0b3a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ebb8f08c89cc3e2a841eef810f0ecfbff47a7d4487cb6075cac2b569fb0b3a8/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/615d5a302f84cae2265ede00172cb57c26bdccd29fac16ef39be19422eb05cfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/615d5a302f84cae2265ede00172cb57c26bdccd29fac16ef39be19422eb05cfc/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6245e620c4fd603bb30da8e0539721bab255b15781717543fee7786d05b7e371/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6245e620c4fd603bb30da8e0539721bab255b15781717543fee7786d05b7e371/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75da86808613f6af8b190bb7c1afd414b89634718109eda5e8bef1d3b66203c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75da86808613f6af8b190bb7c1afd414b89634718109eda5e8bef1d3b66203c1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84e04ea362d6752c70c4ac2f6b3740f33e5d9570ae17d399ee9246731b4fddc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84e04ea362d6752c70c4ac2f6b3740f33e5d9570ae17d399ee9246731b4fddc0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90b58fb6ab9014b5f6e8735e8734de09b20715a122397456737fe29b7e86a1d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90b58fb6ab9014b5f6e8735e8734de09b20715a122397456737fe29b7e86a1d9/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4e122d178826bc4ca4a231963dd65f5e698fa34878685473f5bd1f808c64d32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4e122d178826bc4ca4a231963dd65f5e698fa34878685473f5bd1f808c64d32/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a75c718e93f64cf58132066f8cef04845af0ee72b9270a8a2bf6d00aa18d3adb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a75c718e93f64cf58132066f8cef04845af0ee72b9270a8a2bf6d00aa18d3adb/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b614bb832af3e09b23b359998aca23ab58c5a9ab7952ccb92b54bca9183113d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b614bb832af3e09b23b359998aca23ab58c5a9ab7952ccb92b54bca9183113d4/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1d19ac77f621f194d7f843f3f312fd667cdc82536a41baffa2499562be69330/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1d19ac77f621f194d7f843f3f312fd667cdc82536a41baffa2499562be69330/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3cc2ece358f944d05196764702fea888b194d309b629fc34e79aa1706ea28d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3cc2ece358f944d05196764702fea888b194d309b629fc34e79aa1706ea28d7/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1ab68e70041e210fc7f8c303e709329e0fa1234babd891c1f6a74d78e8bde77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1ab68e70041e210fc7f8c303e709329e0fa1234babd891c1f6a74d78e8bde77/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fdc58d1-11b9-4879-8879-8909f186bbe5/volumes/kubernetes.io~projected/kube-api-access-472cs:{mountpoint:/var/lib/kubelet/pods/1fdc58d1-11b9-4879-8879-8909f186bbe5/volumes/kubernetes.io~projected/kube-api-access-472cs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/268dd18a-b222-4c6d-b981-6babbe21878f/volumes/kubernetes.io~projected/kube-api-access-ltkx6:{mountpoint:/var/lib/kubelet/pods/268dd18a-b222-4c6d-b981-6babbe21878f/volumes/kubernetes.io~projected/kube-api-access-ltkx6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~projected/kube-api-access-6mrs4:{mountpoint:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~projected/kube-api-access-6mrs4 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30369506-d839-4e0b-a652-a3875a88b1b4/volumes/kubernetes.io~projected/kube-api-access-9htbb:{mountpoint:/var/lib/kubelet/pods/30369506-d839-4e0b-a652-a3875a88b1b4/volumes/kubernetes.io~projected/kube-api-access-9htbb major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~projected/kube-api-access-7l8n9:{mountpoint:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~projected/kube-api-access-7l8n9 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4572a1ca-c58c-4207-a080-7dae7263882a/volumes/kubernetes.io~projected/kube-api-access-5ct75:{mountpoint:/var/lib/kubelet/pods/4572a1ca-c58c-4207-a080-7dae7263882a/volumes/kubernetes.io~projected/kube-api-access-5ct75 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volumes/kubernetes.io~projected/kube-api-access-r4hr9:{mountpoint:/var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volumes/kubernetes.io~projected/kube-api-access-r4hr9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4775f051-dc74-47a3-b528-a735daac1287/volumes/kubernetes.io~projected/kube-api-access-6sc26:{mountpoint:/var/lib/kubelet/pods/4775f051-dc74-47a3-b528-a735daac1287/volumes/kubernetes.io~projected/kube-api-access-6sc26 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4775f051-dc74-47a3-b528-a735daac1287/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4775f051-dc74-47a3-b528-a735daac1287/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~projected/kube-api-access-6dssr:{mountpoint:/var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~projected/kube-api-access-6dssr major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4/volumes/kubernetes.io~projected/kube-api-access-czfjj:{mountpoint:/var/lib/kubelet/pods/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4/volumes/kubernetes.io~projected/kube-api-access-czfjj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5719f87e-6553-4207-8b40-766d750d8b4c/volumes/kubernetes.io~projected/kube-api-access-jlzc2:{mountpoint:/var/lib/kubelet/pods/5719f87e-6553-4207-8b40-766d750d8b4c/volumes/kubernetes.io~projected/kube-api-access-jlzc2 major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~projected/kube-api-access-tlcbg:{mountpoint:/var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~projected/kube-api-access-tlcbg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62848175-7827-47e2-af47-7edd2af96935/volumes/kubernetes.io~projected/kube-api-access-f7q47:{mountpoint:/var/lib/kubelet/pods/62848175-7827-47e2-af47-7edd2af96935/volumes/kubernetes.io~projected/kube-api-access-f7q47 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ae2e290-c982-4b8a-a815-d3babf2679ba/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7ae2e290-c982-4b8a-a815-d3babf2679ba/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1/volumes/kubernetes.io~projected/kube-api-access-n9k76:{mountpoint:/var/lib/kubelet/pods/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1/volumes/kubernetes.io~projected/kube-api-access-n9k76 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e/volumes/kubernetes.io~projected/kube-api-access-ns4qp:{mountpoint:/var/lib/kubelet/pods/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e/volumes/kubernetes.io~projected/kube-api-access-ns4qp major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/920770c6-fd18-41d7-95f6-a55c195361e6/volumes/kubernetes.io~projected/kube-api-access-6cr2j:{mountpoint:/var/lib/kubelet/pods/920770c6-fd18-41d7-95f6-a55c195361e6/volumes/kubernetes.io~projected/kube-api-access-6cr2j major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99936bc2-3cbd-445f-9a02-f387cedc1438/volumes/kubernetes.io~projected/kube-api-access-9tww6:{mountpoint:/var/lib/kubelet/pods/99936bc2-3cbd-445f-9a02-f387cedc1438/volumes/kubernetes.io~projected/kube-api-access-9tww6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a549fa1-91e2-4c55-b306-2baf8e54dd96/volumes/kubernetes.io~projected/kube-api-access-4fnmz:{mountpoint:/var/lib/kubelet/pods/9a549fa1-91e2-4c55-b306-2baf8e54dd96/volumes/kubernetes.io~projected/kube-api-access-4fnmz major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9aaad021-a27a-4ee3-97d7-71fbaa1aff87/volumes/kubernetes.io~projected/kube-api-access-gb8js:{mountpoint:/var/lib/kubelet/pods/9aaad021-a27a-4ee3-97d7-71fbaa1aff87/volumes/kubernetes.io~projected/kube-api-access-gb8js major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd08e0e-c491-487a-a4f6-a0955b41c4a0/volumes/kubernetes.io~projected/kube-api-access-nkdxm:{mountpoint:/var/lib/kubelet/pods/9cd08e0e-c491-487a-a4f6-a0955b41c4a0/volumes/kubernetes.io~projected/kube-api-access-nkdxm major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd08e0e-c491-487a-a4f6-a0955b41c4a0/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/9cd08e0e-c491-487a-a4f6-a0955b41c4a0/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c89d90-bbaa-4f37-acb6-c83e561a1d72/volumes/kubernetes.io~projected/kube-api-access-wfxhv:{mountpoint:/var/lib/kubelet/pods/a7c89d90-bbaa-4f37-acb6-c83e561a1d72/volumes/kubernetes.io~projected/kube-api-access-wfxhv major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c89d90-bbaa-4f37-acb6-c83e561a1d72/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a7c89d90-bbaa-4f37-acb6-c83e561a1d72/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcde6ed5-cbad-482d-9f42-b6218e390bd2/volumes/kubernetes.io~projected/kube-api-access-lj8vz:{mountpoint:/var/lib/kubelet/pods/bcde6ed5-cbad-482d-9f42-b6218e390bd2/volumes/kubernetes.io~projected/kube-api-access-lj8vz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf193224-26cc-4cd6-8aa5-2424e1212c2a/volumes/kubernetes.io~projected/kube-api-access-7cmmt:{mountpoint:/var/lib/kubelet/pods/bf193224-26cc-4cd6-8aa5-2424e1212c2a/volumes/kubernetes.io~projected/kube-api-access-7cmmt major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2198f51-8630-484c-9825-bb91b3b897e4/volumes/kubernetes.io~projected/kube-api-access-m42fq:{mountpoint:/var/lib/kubelet/pods/c2198f51-8630-484c-9825-bb91b3b897e4/volumes/kubernetes.io~projected/kube-api-access-m42fq major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d07ee097-855f-4598-a947-2268077bac00/volumes/kubernetes.io~projected/kube-api-access-6ssfj:{mountpoint:/var/lib/kubelet/pods/d07ee097-855f-4598-a947-2268077bac00/volumes/kubernetes.io~projected/kube-api-access-6ssfj major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d07ee097-855f-4598-a947-2268077bac00/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d07ee097-855f-4598-a947-2268077bac00/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8aed736-cadf-4091-b78e-1d555860a055/volumes/kubernetes.io~projected/kube-api-access-mbl8q:{mountpoint:/var/lib/kubelet/pods/d8aed736-cadf-4091-b78e-1d555860a055/volumes/kubernetes.io~projected/kube-api-access-mbl8q major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e84861d0-c9a6-480b-82ed-5a0ce96dab7d/volumes/kubernetes.io~projected/kube-api-access-s2q97:{mountpoint:/var/lib/kubelet/pods/e84861d0-c9a6-480b-82ed-5a0ce96dab7d/volumes/kubernetes.io~projected/kube-api-access-s2q97 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e84861d0-c9a6-480b-82ed-5a0ce96dab7d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e84861d0-c9a6-480b-82ed-5a0ce96dab7d/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66/volumes/kubernetes.io~projected/kube-api-access-pc6rv:{mountpoint:/var/lib/kubelet/pods/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66/volumes/kubernetes.io~projected/kube-api-access-pc6rv major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0/volumes/kubernetes.io~projected/kube-api-access-scwm4:{mountpoint:/var/lib/kubelet/pods/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0/volumes/kubernetes.io~projected/kube-api-access-scwm4 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0/volumes/kubernetes.io~secret/cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f00a23f9-5c7e-437b-929c-4f793b200fca/volumes/kubernetes.io~projected/kube-api-access-plqkh:{mountpoint:/var/lib/kubelet/pods/f00a23f9-5c7e-437b-929c-4f793b200fca/volumes/kubernetes.io~projected/kube-api-access-plqkh major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82/volumes/kubernetes.io~projected/kube-api-access-gfzbf:{mountpoint:/var/lib/kubelet/pods/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82/volumes/kubernetes.io~projected/kube-api-access-gfzbf major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/84ed147b35536a781a8d9cd16b171f2696af05dac2d98fd2f449dcfaafe989ea/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cec61c67e067756173526c3940248bc8df0dd3aba76c8b3ff9d456f3cec8e27b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/507076de073fb5f7affb520729d1306c07e8f2abdd438c49aa2e37f1df270ad6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/333ace63248423532bed3bac3ce9d5659fd8a89d6aee6a380cd645b3419c5984/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/40b9813128ef6c2b3547f7c4e80e427cdb32ae17be7cdea521998c1382518ed0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ea85fd3cbca8a27e88b3251498d654e38e3f9af08e7be12eea135fc36b591070/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8559a7722f6a6d006a8b4dd0ed41b2bcef16f15a6c8989f5b060addfb637a23a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1a0b9b0ce316027feda1365a3bdd91a83d076f0ded9990e7b8f51360cd92bc87/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/422688f258d1f9422c8d6eafff89864e5825224bf16a987fe280342579b6e38f/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/72d6eca9184c1d2ab107d859442ee559e75b32f6bf085ca64f631b583aee78c4/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/595c4c4d66e41733e9995838e63f2120ce843de89b65569b3f867046505f219c/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3546e21bd2825de3eedc68d1e12ad51125d437e8d395384373d1fb27b2f7d542/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/65c2ae8fdbc27a20d90cec42ead9494446e5eb5ddd3f8d503203fdefc6459fd0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8a35e3329e2855b74fc6b418f66dd1f4b3bec3c7903e658dad9797329f985ea8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bed019143e3c5d1eb68462156bf83870a970ad0f146b03eab3679d95dc0ffcb9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/473ccc803039cbb9673f274ba02e7dfdc1c8ce458f9e0dacb1f43fc48e70483d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8f97420655fe52df9de336ca232bad3bfdb09accde16529a8f2cca4b55c7d6d8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/97c31303674980e3b0ddefa18ea599c968ff831fa5d5445ab35c922136afaab1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4414931910da79458e63450ddadd14b89017fcf2bba9694d8ee50a9985ab986e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a5da12ae372a5abc8c6377d736f669124fa26305bbc572d50e5ed7aea554fc95/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a47f642bbf3b1862815a73c7ea997bb89d7da79083547b5068ebd03d763675b9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/099185d2bd21beb543729b3c65c41e2e2a78d85a6388857a5c7f3e92eb323ad3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/4dfee114d29f4f9cae6d68596162d78f50bef1ac836a5985c83f9cdccb0e4ec7/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/72e694bfb056940da68c81588cd50858bf25ed89553bd35c7f243440ec468fc0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/8c57232f7cc429e49229538a7e5f50893f7653c3740ba8350a404893ee231496/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/47bcd1e567305e68be34364c40c63c50525e5b1d5b6c585e39e337e0e3bfb2a5/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/a13ae51547feee02d312e869af0b5cb9ec5aa1af607cfea35fc6cf897d8b208b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/2bc64fbe59e1117768dc00a1fd5b6317372423bd3200e7b4b21b189252faea7c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/7e63f10e9c74274ffea660fdde34488527ade5eecae9f4983f40f9cea58fd579/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/38868b3ddf04c439e072fc1904dbda7e7f8dae7590dc140cebb20f562a3a99a0/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/77db7227609a7b45576776cbdfb5500e12ff000207f7474c2af5b379bfd020dd/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/da0aa13737327dd788815681d79e01cb6c37b5e53a22c3d1c84d3644efeb4408/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/a1eda63865fdbb9b825ac32dfead350c0ea6f1d995a10e104fbd793fb3370f62/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/fbbf2f15518ed359f62be7710cec0f59b4ae8390b9c7fea1c639ab740d6aef6a/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/8c55e8104f69694e684ec1ea44581918cb8fdebc7964d0a6aab350276d4cfb7c/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3d91b50efdbd7f60a545fae6fc7f95edc8728d383361e9b633b54ebd2e587fe5/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/c5092612cef7d0f37fadceaf0c36c28455457ad0504d874fb4eec21bb3fe4b1a/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/751b90f048e663ccae49c1211890d414d33ac897f0344c447ac91a142b952d40/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/67484893aecbd59575a25f776449b0bcb2c2c0dee56adc3239135d93cd36b52c/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/38f7df0f66b99eb473709f872f98793eae767393b0f177e41a5975a0a1eace5f/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/140a82b7106f7ecd39dec73934245e27b60d2fa20a071fcdbe8877d7935b8890/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/f2c5245851f87d946f6e9eccfcac056bb21b627facffa9539e5e70ae1553dd16/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/8e4b1ec641e58150a6ede239f845a178cbad451f4435f428cc4354ee2ebdad9e/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/0cd3bd170b7fba27ffed2c9167a2e510c41eb5305f7eae28d43d41cdcda816b8/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/74da39ed2bd68051689d284986ca5e7e92378230cc8188b8d99e8e317a42db58/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/e55d6f5a61e26f49afd1cbd5bddb3608d575146b8b881f23337cbe08314c9c87/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/a0b0d544de826795169fdaf53d411aab21c43b318ae46b448fafd2c29563cef3/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/11a61d57361f9082d933b86204d9004b499c8c41343bc8782a68575e7ae0ccb9/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/9bdb516a486834b86607b47dca878bb49da548e1871be89cd608244940a5321b/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/3bc0d542dc242d3f154eb2ef8088550e1699c8f9302645aa116b30a0a3244ed6/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fea4cebff06dcbd0580aef873ff5ed8ac1f413b599e7d3758982f39bd44b6892/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/726f04294181e410a06d95bc9c39f932b3f56de8cb5e16f968f2af9914f17ad3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/53fbc00983c1c95252841af07ba3bbd67abe3e907f8c34623d875242f930003c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/553175dac96e205ba8e5a3ada378851bb0125adfd7dd0b5316defa7a5a98e2ce/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/5697d4d074118aa4e30e9918265c4966b7d4e3796370a998b23b356f0e55eeb2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e324c3c31c26b3fed34c91b364bbd032ffc15a49f516e57cf69e93e94c39e424/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/44511b5526e7011940eaf3cf2874d915d104744deafb64a332ebb3da2060c488/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/5080b9a8ad6a4d53aa0beb8a549bb602594956437a4f2550d1f83b1543f84cc3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/da7dab5e98379e1a07db1fe5fcf19be1fec81823f8bf213fa1b0a51d2d8a1f1c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c9b99f129168f7df3e7d6d1713abb56369545c80c0ac4eae0f716f3dc1e35a4a/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:49:48.559749 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.557006    5031 manager.go:217] Machine: {Timestamp:2026-10-30 21:49:48.556310555 +0000 UTC m=+0.080453310 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199571 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258c07aff4542c531a08703f6f1208 SystemUUID:ec258c07-aff4-542c-531a-08703f6f1208 BootID:df969df4-f327-4abf-aa3a-057a4d6feac8 Filesystems:[{Device:/var/lib/kubelet/pods/a7c89d90-bbaa-4f37-acb6-c83e561a1d72/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd08e0e-c491-487a-a4f6-a0955b41c4a0/volumes/kubernetes.io~projected/kube-api-access-nkdxm DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4/volumes/kubernetes.io~projected/kube-api-access-czfjj DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/589b48d1061ddbd25326b757f7cf01fc11c79ffe22b7f9d3369b9cb9f0ad77f3/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6245e620c4fd603bb30da8e0539721bab255b15781717543fee7786d05b7e371/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/615d5a302f84cae2265ede00172cb57c26bdccd29fac16ef39be19422eb05cfc/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1/volumes/kubernetes.io~projected/kube-api-access-n9k76 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99936bc2-3cbd-445f-9a02-f387cedc1438/volumes/kubernetes.io~projected/kube-api-access-9tww6 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55e1f0b2c6b61cd08c854054a7562c94b1d1a8a8b042ac432a7e876805dcb5c2/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/920770c6-fd18-41d7-95f6-a55c195361e6/volumes/kubernetes.io~projected/kube-api-access-6cr2j DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d07ee097-855f-4598-a947-2268077bac00/volumes/kubernetes.io~projected/kube-api-access-6ssfj DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3cc2ece358f944d05196764702fea888b194d309b629fc34e79aa1706ea28d7/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75da86808613f6af8b190bb7c1afd414b89634718109eda5e8bef1d3b66203c1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2763a23ba894a416b6d0c233ca992372f718767485552d8505aaedeee1f8ef55/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30369506-d839-4e0b-a652-a3875a88b1b4/volumes/kubernetes.io~projected/kube-api-access-9htbb DeviceMajor:0 DeviceMinor:351 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1ab68e70041e210fc7f8c303e709329e0fa1234babd891c1f6a74d78e8bde77/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9a549fa1-91e2-4c55-b306-2baf8e54dd96/volumes/kubernetes.io~projected/kube-api-access-4fnmz DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4e122d178826bc4ca4a231963dd65f5e698fa34878685473f5bd1f808c64d32/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4572a1ca-c58c-4207-a080-7dae7263882a/volumes/kubernetes.io~projected/kube-api-access-5ct75 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bfdbbadcc4984a12e470900f67e8c5b696bbd60975f278a5a16876aa76d3217/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~projected/kube-api-access-6mrs4 DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b614bb832af3e09b23b359998aca23ab58c5a9ab7952ccb92b54bca9183113d4/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/468bd96adaec12e1b28d6e1a218f3e03184c2a84f59b7c71df0cf693d0ed46af/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fdc58d1-11b9-4879-8879-8909f186bbe5/volumes/kubernetes.io~projected/kube-api-access-472cs DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9aaad021-a27a-4ee3-97d7-71fbaa1aff87/volumes/kubernetes.io~projected/kube-api-access-gb8js DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/268dd18a-b222-4c6d-b981-6babbe21878f/volumes/kubernetes.io~projected/kube-api-access-ltkx6 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ebb8f08c89cc3e2a841eef810f0ecfbff47a7d4487cb6075cac2b569fb0b3a8/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84e04ea362d6752c70c4ac2f6b3740f33e5d9570ae17d399ee9246731b4fddc0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4775f051-dc74-47a3-b528-a735daac1287/volumes/kubernetes.io~projected/kube-api-access-6sc26 DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8aed736-cadf-4091-b78e-1d555860a055/volumes/kubernetes.io~projected/kube-api-access-mbl8q DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a75c718e93f64cf58132066f8cef04845af0ee72b9270a8a2bf6d00aa18d3adb/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/255554e0a0dbb590af06528ee7c1b20b7dc96950b2e39b104188f2f0d9de5482/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dcc139cffd7b6c9b5f1e3d558d85a519322c05cde86d44a84cdbb44e2fa21e4/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~projected/kube-api-access-6dssr DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e/volumes/kubernetes.io~projected/kube-api-access-ns4qp DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0/volumes/kubernetes.io~projected/kube-api-access-scwm4 DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f00a23f9-5c7e-437b-929c-4f793b200fca/volumes/kubernetes.io~projected/kube-api-access-plqkh DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66/volumes/kubernetes.io~projected/kube-api-access-pc6rv DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a6789d1-ca4c-4efe-b973-8c764ae03b89/volumes/kubernetes.io~projected/kube-api-access-tlcbg DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf193224-26cc-4cd6-8aa5-2424e1212c2a/volumes/kubernetes.io~projected/kube-api-access-7cmmt DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90b58fb6ab9014b5f6e8735e8734de09b20715a122397456737fe29b7e86a1d9/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62848175-7827-47e2-af47-7edd2af96935/volumes/kubernetes.io~projected/kube-api-access-f7q47 DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcde6ed5-cbad-482d-9f42-b6218e390bd2/volumes/kubernetes.io~projected/kube-api-access-lj8vz DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82/volumes/kubernetes.io~projected/kube-api-access-gfzbf DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ed92975ef9d76ec301f851358b9aa6481d0fcfe8c10ff696833a843d32f5ed5/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volumes/kubernetes.io~projected/kube-api-access-r4hr9 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2587c6a50a557f5c522cb7d8c1589baba2d4e84bee219796d490ee6b2c4b23bd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1d19ac77f621f194d7f843f3f312fd667cdc82536a41baffa2499562be69330/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e84861d0-c9a6-480b-82ed-5a0ce96dab7d/volumes/kubernetes.io~projected/kube-api-access-s2q97 DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd08e0e-c491-487a-a4f6-a0955b41c4a0/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ae2e290-c982-4b8a-a815-d3babf2679ba/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ea14ac24914466f3ff350290786b2fb621a3ec9b831e25d78f49b740ee590ba/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes/kubernetes.io~projected/kube-api-access-7l8n9 DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4aa635da86fa6866138ac0a2957518a7984ddd913ef3e1c5fe487c13ef15ccf8/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d07ee097-855f-4598-a947-2268077bac00/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c89d90-bbaa-4f37-acb6-c83e561a1d72/volumes/kubernetes.io~projected/kube-api-access-wfxhv DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5719f87e-6553-4207-8b40-766d750d8b4c/volumes/kubernetes.io~projected/kube-api-access-jlzc2 DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4775f051-dc74-47a3-b528-a735daac1287/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47bc5387-3f28-4d19-9f51-cdd6e298896f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2198f51-8630-484c-9825-bb91b3b897e4/volumes/kubernetes.io~projected/kube-api-access-m42fq DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45bbb59a-b1e7-4993-8a0b-b82f54b1c844/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e84861d0-c9a6-480b-82ed-5a0ce96dab7d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00c6e38ab33fb7a MacAddress:9e:8c:55:ff:39:92 Speed:10000 Mtu:8901} {Name:1ed92975ef9d76e MacAddress:d2:1b:4c:72:ac:e1 Speed:10000 Mtu:8901} {Name:255554e0a0dbb59 MacAddress:2a:33:bb:b8:b7:d4 Speed:10000 Mtu:8901} {Name:2763a23ba894a41 MacAddress:1a:6f:3e:be:20:a4 Speed:10000 Mtu:8901} {Name:3ea14ac24914466 MacAddress:2e:35:69:6d:ee:3c Speed:10000 Mtu:8901} {Name:468bd96adaec12e MacAddress:b6:2d:c6:f9:4e:37 Speed:10000 Mtu:8901} {Name:589b48d1061ddbd MacAddress:aa:05:0d:5d:02:9e Speed:10000 Mtu:8901} {Name:615d5a302f84cae MacAddress:52:d5:a3:cf:27:09 Speed:10000 Mtu:8901} {Name:6245e620c4fd603 MacAddress:c2:f7:43:b3:e3:11 Speed:10000 Mtu:8901} {Name:90b58fb6ab9014b MacAddress:ee:85:79:38:85:72 Speed:10000 Mtu:8901} {Name:a4e122d178826bc MacAddress:e6:4f:9e:1d:de:b0 Speed:10000 Mtu:8901} {Name:a75c718e93f64cf MacAddress:ce:2a:ae:60:20:9a Speed:10000 Mtu:8901} {Name:b614bb832af3e09 MacAddress:66:c4:fd:55:0c:56 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:51:90:22:e9:61 Speed:0 Mtu:9001} {Name:br-int MacAddress:d2:5f:d6:72:e5:0b Speed:0 Mtu:8901} {Name:e1d19ac77f621f1 MacAddress:06:20:61:f2:fe:42 Speed:10000 Mtu:8901} {Name:e3cc2ece358f944 MacAddress:36:88:d4:11:9b:19 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:51:90:22:e9:61 Speed:0 Mtu:9001} {Name:f1ab68e70041e21 MacAddress:4e:2f:59:ce:a8:ad Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0a:0f:98:6f:ec:3f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:98:e6:c8:e0:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:49:48.559749 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.557552    5031 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:49:48.559749 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.557616    5031 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:49:48.562283 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562256    5031 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:49:48.562420 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562284    5031 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-147.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:49:48.562487 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562421    5031 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:49:48.562487 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562430    5031 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:49:48.562487 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562443    5031 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:48.562487 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562461    5031 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:48.562487 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562479    5031 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:49:48.562679 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562631    5031 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:49:48.562737 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562728    5031 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:49:48.562780 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562760    5031 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:49:48.562780 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562773    5031 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:49:48.562852 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562782    5031 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:49:48.562852 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562791    5031 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:49:48.562852 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.562799    5031 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:49:48.563756 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.563736    5031 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:49:48.563756 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.563752    5031 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:48.566653 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.566634    5031 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:49:48.578629 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.578608    5031 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:49:48.578629 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.578624    5031 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:48.578737 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.578730    5031 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:49:48.578950 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.578937    5031 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:49:48.578991 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.578968    5031 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:49:48.579077 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579062    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:49:48.579122 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579083    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:49:48.579122 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579094    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:49:48.579190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579136    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:49:48.579190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579149    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:49:48.579190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579161    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:49:48.579190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579172    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:49:48.579190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579186    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:49:48.579414 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579201    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:49:48.579414 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579233    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:49:48.579414 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579249    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:49:48.579414 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579259    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:49:48.579414 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579286    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:49:48.579414 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.579293    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:49:48.583242 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.583202    5031 server.go:1263] "Started kubelet"
Oct 30 21:49:48.583359 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.583330    5031 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:49:48.583446 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.583387    5031 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:49:48.583517 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.583473    5031 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:49:48.583867 ip-10-0-141-147 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:49:48.584172 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.584120    5031 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:49:48.588540 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.586489    5031 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:49:48.589713 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.589695    5031 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:49:48.589912 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.589895    5031 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:48.590020 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.590000    5031 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:08 +0000 UTC" deadline="2028-08-15 02:01:27.821900186 +0000 UTC"
Oct 30 21:49:48.590078 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.590021    5031 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15700h11m39.231882007s"
Oct 30 21:49:48.590276 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.590200    5031 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:49:48.590361 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.590349    5031 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:49:48.590510 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.590499    5031 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:49:48.590745 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:48.590708    5031 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:48.591826 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.591806    5031 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:49:48.591826 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.591826    5031 factory.go:55] Registering systemd factory
Oct 30 21:49:48.592009 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.591837    5031 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.592473    5031 factory.go:153] Registering CRI-O factory
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.592490    5031 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.592506    5031 factory.go:103] Registering Raw factory
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.592517    5031 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.592876    5031 manager.go:319] Starting recovery of all containers
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593569    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45bbb59a-b1e7-4993-8a0b-b82f54b1c844" volumeName="kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-config" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593603    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" volumeName="kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593618    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="268dd18a-b222-4c6d-b981-6babbe21878f" volumeName="kubernetes.io/configmap/268dd18a-b222-4c6d-b981-6babbe21878f-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593629    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e458ae3-5f11-4650-ae5e-0145b063be66" volumeName="kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593640    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45bbb59a-b1e7-4993-8a0b-b82f54b1c844" volumeName="kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-env-overrides" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593651    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd08e0e-c491-487a-a4f6-a0955b41c4a0" volumeName="kubernetes.io/configmap/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-cabundle" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593664    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45bbb59a-b1e7-4993-8a0b-b82f54b1c844" volumeName="kubernetes.io/projected/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-kube-api-access-r4hr9" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593676    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47bc5387-3f28-4d19-9f51-cdd6e298896f" volumeName="kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-stats-auth" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593691    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a6789d1-ca4c-4efe-b973-8c764ae03b89" volumeName="kubernetes.io/projected/5a6789d1-ca4c-4efe-b973-8c764ae03b89-kube-api-access-tlcbg" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593702    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2198f51-8630-484c-9825-bb91b3b897e4" volumeName="kubernetes.io/empty-dir/c2198f51-8630-484c-9825-bb91b3b897e4-tmp-dir" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593713    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d07ee097-855f-4598-a947-2268077bac00" volumeName="kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-trusted-ca" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593725    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e84861d0-c9a6-480b-82ed-5a0ce96dab7d" volumeName="kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-tmp" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593736    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30369506-d839-4e0b-a652-a3875a88b1b4" volumeName="kubernetes.io/projected/30369506-d839-4e0b-a652-a3875a88b1b4-kube-api-access-9htbb" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593747    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdc58d1-11b9-4879-8879-8909f186bbe5" volumeName="kubernetes.io/projected/1fdc58d1-11b9-4879-8879-8909f186bbe5-kube-api-access-472cs" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593760    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e458ae3-5f11-4650-ae5e-0145b063be66" volumeName="kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593768    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e458ae3-5f11-4650-ae5e-0145b063be66" volumeName="kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593777    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d07ee097-855f-4598-a947-2268077bac00" volumeName="kubernetes.io/secret/d07ee097-855f-4598-a947-2268077bac00-serving-cert" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593792    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66" volumeName="kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593800    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e458ae3-5f11-4650-ae5e-0145b063be66" volumeName="kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593809    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45bbb59a-b1e7-4993-8a0b-b82f54b1c844" volumeName="kubernetes.io/secret/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593820    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62848175-7827-47e2-af47-7edd2af96935" volumeName="kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593827    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd08e0e-c491-487a-a4f6-a0955b41c4a0" volumeName="kubernetes.io/projected/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-kube-api-access-nkdxm" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593834    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c89d90-bbaa-4f37-acb6-c83e561a1d72" volumeName="kubernetes.io/secret/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-serving-cert" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593843    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8aed736-cadf-4091-b78e-1d555860a055" volumeName="kubernetes.io/empty-dir/d8aed736-cadf-4091-b78e-1d555860a055-data-volume" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593852    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4572a1ca-c58c-4207-a080-7dae7263882a" volumeName="kubernetes.io/projected/4572a1ca-c58c-4207-a080-7dae7263882a-kube-api-access-5ct75" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593860    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45bbb59a-b1e7-4993-8a0b-b82f54b1c844" volumeName="kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593867    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59dafadb-fce7-4a75-afc2-fc4275882723" volumeName="kubernetes.io/configmap/59dafadb-fce7-4a75-afc2-fc4275882723-nginx-conf" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593875    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99936bc2-3cbd-445f-9a02-f387cedc1438" volumeName="kubernetes.io/projected/99936bc2-3cbd-445f-9a02-f387cedc1438-kube-api-access-9tww6" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593884    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" volumeName="kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593894    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" volumeName="kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593907    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62848175-7827-47e2-af47-7edd2af96935" volumeName="kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593917    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae2e290-c982-4b8a-a815-d3babf2679ba" volumeName="kubernetes.io/secret/7ae2e290-c982-4b8a-a815-d3babf2679ba-agent-certs" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593926    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82" volumeName="kubernetes.io/projected/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-kube-api-access-gfzbf" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593935    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" volumeName="kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593953    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47bc5387-3f28-4d19-9f51-cdd6e298896f" volumeName="kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-default-certificate" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593963    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" volumeName="kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593972    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcde6ed5-cbad-482d-9f42-b6218e390bd2" volumeName="kubernetes.io/projected/bcde6ed5-cbad-482d-9f42-b6218e390bd2-kube-api-access-lj8vz" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593980    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f00a23f9-5c7e-437b-929c-4f793b200fca" volumeName="kubernetes.io/projected/f00a23f9-5c7e-437b-929c-4f793b200fca-kube-api-access-plqkh" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593987    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" volumeName="kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.593997    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4775f051-dc74-47a3-b528-a735daac1287" volumeName="kubernetes.io/configmap/4775f051-dc74-47a3-b528-a735daac1287-config" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594019    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a6789d1-ca4c-4efe-b973-8c764ae03b89" volumeName="kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-tuned" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594027    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d07ee097-855f-4598-a947-2268077bac00" volumeName="kubernetes.io/projected/d07ee097-855f-4598-a947-2268077bac00-kube-api-access-6ssfj" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594034    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82" volumeName="kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594046    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" volumeName="kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594053    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" volumeName="kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594059    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5719f87e-6553-4207-8b40-766d750d8b4c" volumeName="kubernetes.io/projected/5719f87e-6553-4207-8b40-766d750d8b4c-kube-api-access-jlzc2" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594066    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c89d90-bbaa-4f37-acb6-c83e561a1d72" volumeName="kubernetes.io/projected/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-kube-api-access-wfxhv" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594074    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d07ee097-855f-4598-a947-2268077bac00" volumeName="kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-config" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594080    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8aed736-cadf-4091-b78e-1d555860a055" volumeName="kubernetes.io/projected/d8aed736-cadf-4091-b78e-1d555860a055-kube-api-access-mbl8q" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594089    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e84861d0-c9a6-480b-82ed-5a0ce96dab7d" volumeName="kubernetes.io/secret/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-serving-cert" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594096    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" volumeName="kubernetes.io/projected/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-kube-api-access-czfjj" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594103    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62848175-7827-47e2-af47-7edd2af96935" volumeName="kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594111    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f32ae7a-4fbd-4959-a49b-76e9729a5c8e" volumeName="kubernetes.io/configmap/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-config" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594118    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf193224-26cc-4cd6-8aa5-2424e1212c2a" volumeName="kubernetes.io/projected/bf193224-26cc-4cd6-8aa5-2424e1212c2a-kube-api-access-7cmmt" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594125    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2198f51-8630-484c-9825-bb91b3b897e4" volumeName="kubernetes.io/configmap/c2198f51-8630-484c-9825-bb91b3b897e4-config-volume" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594133    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e458ae3-5f11-4650-ae5e-0145b063be66" volumeName="kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594147    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5" volumeName="kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594154    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" volumeName="kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594161    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4572a1ca-c58c-4207-a080-7dae7263882a" volumeName="kubernetes.io/empty-dir/4572a1ca-c58c-4207-a080-7dae7263882a-tmp-dir" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594168    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47bc5387-3f28-4d19-9f51-cdd6e298896f" volumeName="kubernetes.io/projected/47bc5387-3f28-4d19-9f51-cdd6e298896f-kube-api-access-6dssr" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594175    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9aaad021-a27a-4ee3-97d7-71fbaa1aff87" volumeName="kubernetes.io/projected/9aaad021-a27a-4ee3-97d7-71fbaa1aff87-kube-api-access-gb8js" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594184    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd08e0e-c491-487a-a4f6-a0955b41c4a0" volumeName="kubernetes.io/secret/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-key" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594193    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c89d90-bbaa-4f37-acb6-c83e561a1d72" volumeName="kubernetes.io/configmap/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-config" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594200    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="268dd18a-b222-4c6d-b981-6babbe21878f" volumeName="kubernetes.io/projected/268dd18a-b222-4c6d-b981-6babbe21878f-kube-api-access-ltkx6" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594226    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62848175-7827-47e2-af47-7edd2af96935" volumeName="kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594239    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf193224-26cc-4cd6-8aa5-2424e1212c2a" volumeName="kubernetes.io/configmap/bf193224-26cc-4cd6-8aa5-2424e1212c2a-telemetry-config" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594251    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2198f51-8630-484c-9825-bb91b3b897e4" volumeName="kubernetes.io/projected/c2198f51-8630-484c-9825-bb91b3b897e4-kube-api-access-m42fq" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594265    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e84861d0-c9a6-480b-82ed-5a0ce96dab7d" volumeName="kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-snapshots" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594278    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaec4c99-a4dd-47e4-9f3d-f6a276f166a0" volumeName="kubernetes.io/projected/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-kube-api-access-scwm4" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594289    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e458ae3-5f11-4650-ae5e-0145b063be66" volumeName="kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594302    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a6789d1-ca4c-4efe-b973-8c764ae03b89" volumeName="kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-tmp" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594314    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a549fa1-91e2-4c55-b306-2baf8e54dd96" volumeName="kubernetes.io/projected/9a549fa1-91e2-4c55-b306-2baf8e54dd96-kube-api-access-4fnmz" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594356    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcde6ed5-cbad-482d-9f42-b6218e390bd2" volumeName="kubernetes.io/configmap/bcde6ed5-cbad-482d-9f42-b6218e390bd2-serviceca" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594374    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82" volumeName="kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594386    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e458ae3-5f11-4650-ae5e-0145b063be66" volumeName="kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594399    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" volumeName="kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594410    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4775f051-dc74-47a3-b528-a735daac1287" volumeName="kubernetes.io/projected/4775f051-dc74-47a3-b528-a735daac1287-kube-api-access-6sc26" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594421    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4775f051-dc74-47a3-b528-a735daac1287" volumeName="kubernetes.io/secret/4775f051-dc74-47a3-b528-a735daac1287-serving-cert" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594432    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae2e290-c982-4b8a-a815-d3babf2679ba" volumeName="kubernetes.io/configmap/7ae2e290-c982-4b8a-a815-d3babf2679ba-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594444    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="920770c6-fd18-41d7-95f6-a55c195361e6" volumeName="kubernetes.io/projected/920770c6-fd18-41d7-95f6-a55c195361e6-kube-api-access-6cr2j" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594455    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e84861d0-c9a6-480b-82ed-5a0ce96dab7d" volumeName="kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594467    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e84861d0-c9a6-480b-82ed-5a0ce96dab7d" volumeName="kubernetes.io/projected/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-kube-api-access-s2q97" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594478    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f32ae7a-4fbd-4959-a49b-76e9729a5c8e" volumeName="kubernetes.io/projected/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-kube-api-access-ns4qp" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594489    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e84861d0-c9a6-480b-82ed-5a0ce96dab7d" volumeName="kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:49:48.594433 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594502    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1" volumeName="kubernetes.io/projected/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kube-api-access-n9k76" seLinuxMountContext=""
Oct 30 21:49:48.597827 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594513    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaec4c99-a4dd-47e4-9f3d-f6a276f166a0" volumeName="kubernetes.io/secret/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-cert" seLinuxMountContext=""
Oct 30 21:49:48.597827 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594525    5031 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:49:48.597827 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.594531    5031 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:49:48.606049 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.603780    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:49:48.609959 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.609923    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:49:48.635386 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.635295    5031 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:49:48.637176 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.637146    5031 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:49:48.637326 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.637194    5031 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:49:48.637326 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.637269    5031 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:49:48.637326 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:48.637313    5031 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:49:48.676120 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.676082    5031 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="56a5f72177009507be1b949fa056f6d008eafc0b4902a0423ae35e38fe991160" exitCode=0
Oct 30 21:49:48.676120 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.676116    5031 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="023e0f1ad6bdc5e06c2eaeae4c2f189a0f1d94aa1fc37977aae8847c150f53bf" exitCode=0
Oct 30 21:49:48.676120 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.676125    5031 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="928c4206450b07636e91ea3fed57ceafacdb27f354fb5910fbc7dc4647dafc7a" exitCode=0
Oct 30 21:49:48.676402 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.676136    5031 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="74b0ea871ccf0614a6e56e46dfce62525c929732448a46f8c7371ef1f180cf1d" exitCode=0
Oct 30 21:49:48.676402 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.676144    5031 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="531e04af24a5f0a0aa8658b765902260bb3e8dd6ffa6736d2fad0680957d3812" exitCode=0
Oct 30 21:49:48.676402 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.676151    5031 generic.go:356] "Generic (PLEG): container finished" podID="56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4" containerID="b1a5035a137b2a4474009d2a6e2892d49395279a6024f81c08947624bd5712f4" exitCode=0
Oct 30 21:49:48.692477 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:48.692447    5031 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:48.709249 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.709194    5031 generic.go:356] "Generic (PLEG): container finished" podID="e2135f0c5d9f6a303bc10153d13e438d" containerID="009bf78a94203c34d2031d5015b0a6df28756d15b7e78929ff2273f329a997e4" exitCode=0
Oct 30 21:49:48.711009 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.710989    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:49:48.711123 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.711017    5031 generic.go:356] "Generic (PLEG): container finished" podID="d07ee097-855f-4598-a947-2268077bac00" containerID="0124511bc3d9d40471df983f4e5b72596bc0e86b208962998ff187d6d35cf3be" exitCode=255
Oct 30 21:49:48.712412 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.712397    5031 manager.go:324] Recovery completed
Oct 30 21:49:48.734766 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.734747    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:48.737380 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.737347    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:48.737482 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.737385    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:48.737482 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.737396    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:48.737482 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:48.737348    5031 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:49:48.740014 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.739995    5031 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:49:48.740014 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740009    5031 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:49:48.740152 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740018    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:48.740152 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740024    5031 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:49:48.740244 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740231    5031 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:49:48.740274 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740243    5031 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:49:48.740274 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740259    5031 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:49:48.740274 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740265    5031 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:49:48.740274 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740270    5031 policy_none.go:50] "Start"
Oct 30 21:49:48.740382 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740278    5031 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:49:48.740382 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740288    5031 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:49:48.740436 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740410    5031 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:49:48.740436 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740417    5031 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:49:48.740436 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740422    5031 policy_none.go:44] "Start"
Oct 30 21:49:48.740861 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740846    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:48.740938 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740873    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:48.740938 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.740886    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:48.744747 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.744723    5031 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:49:48.748130 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748115    5031 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:49:48.748223 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:48.748138    5031 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:49:48.748223 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748147    5031 server.go:87] "Starting device plugin registration server"
Oct 30 21:49:48.748412 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748391    5031 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:49:48.748467 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748414    5031 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:49:48.748578 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748566    5031 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:49:48.748663 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748632    5031 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:49:48.748724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748667    5031 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:49:48.748724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748673    5031 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:49:48.749407 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.748712    5031 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:49:48.7486596Z","UUID":"bf60d23a-65fd-4205-b01f-3704f30ab058","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:49:48.751249 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.750520    5031 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:49:48.751249 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.750558    5031 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:49:48.751249 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:48.750555    5031 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:49:48.751249 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:48.750637    5031 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-147.us-west-1.compute.internal\" not found"
Oct 30 21:49:48.798025 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.798002    5031 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:48.798136 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.798028    5031 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:48.798136 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.798075    5031 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:49:48.848648 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.848627    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:48.849576 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.849561    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:48.849662 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.849590    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:48.849662 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.849600    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-147.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:48.849662 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.849644    5031 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.850115 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.850099    5031 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:48.850514 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.850501    5031 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:48.938329 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.938241    5031 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"]
Oct 30 21:49:48.938995 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.938947    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"e2135f0c5d9f6a303bc10153d13e438d","Type":"ContainerStarted","Data":"7a6c237ebfc373c7799e27fc55c55b2ca8aaf35ba0027ebc8ebb3c0542d6bd3d"}
Oct 30 21:49:48.939094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.939000    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"e2135f0c5d9f6a303bc10153d13e438d","Type":"ContainerDied","Data":"009bf78a94203c34d2031d5015b0a6df28756d15b7e78929ff2273f329a997e4"}
Oct 30 21:49:48.939094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.939019    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"e2135f0c5d9f6a303bc10153d13e438d","Type":"ContainerStarted","Data":"5919737aea89ad04a291dd2f689e002763130d7f77943e329ce6cfcff81cc8e4"}
Oct 30 21:49:48.939094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.939047    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.939094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.939057    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"46aa86434e4345d68e5cda43f3103b52","Type":"ContainerStarted","Data":"9101a98786086577667bf20840da1426d13d0f6cf5d3dd1b936e84f8c21c34c8"}
Oct 30 21:49:48.939094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.939076    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal" event={"ID":"46aa86434e4345d68e5cda43f3103b52","Type":"ContainerStarted","Data":"2a6db8a5010297c5aec71772aa327dabc74a5bf9037f6f06c9a6b979d389ca36"}
Oct 30 21:49:48.939277 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.939103    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.965852 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.965827    5031 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.965948 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.965887    5031 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.999000 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.998966    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.999121 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.999005    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:48.999121 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:48.999029    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46aa86434e4345d68e5cda43f3103b52-config\") pod \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"46aa86434e4345d68e5cda43f3103b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.015070 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.015042    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:49.015201 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.015092    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:49.015201 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.015135    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.015201 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.015091    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.099575 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.099538    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.099575 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.099569    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.099794 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.099585    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46aa86434e4345d68e5cda43f3103b52-config\") pod \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"46aa86434e4345d68e5cda43f3103b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.099794 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.099630    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46aa86434e4345d68e5cda43f3103b52-config\") pod \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"46aa86434e4345d68e5cda43f3103b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.099794 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.099641    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.099794 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.099653    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2135f0c5d9f6a303bc10153d13e438d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" (UID: \"e2135f0c5d9f6a303bc10153d13e438d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.564119 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.564089    5031 apiserver.go:52] "Watching apiserver"
Oct 30 21:49:49.726417 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.726386    5031 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:49:49.727366 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.727336    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xh5b2","kube-system/global-pull-secret-syncer-jl8q2","openshift-network-diagnostics/network-check-target-cjd7c","openshift-network-operator/iptables-alerter-29pdk","openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh","openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc","kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal","openshift-insights/periodic-gathering-scxcw-zq8dc","openshift-multus/multus-additional-cni-plugins-dwxtj","openshift-ingress/router-default-d48b6d977-dstqd","openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s","openshift-network-console/networking-console-plugin-86c478bb88-g64qk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r","openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp","kube-system/kas-connection-checker-685895c7c9-jqx4n","openshift-ovn-kubernetes/ovnkube-node-8h6kj","kube-system/kas-connection-checker-685895c7c9-m87n9","kube-system/kas-connection-checker-685895c7c9-w9w7t","openshift-dns/node-resolver-q5tqg","openshift-cluster-node-tuning-operator/tuned-rcxbn","openshift-multus/network-metrics-daemon-h8h85","openshift-image-registry/image-registry-8698788495-986ld","openshift-ingress-canary/ingress-canary-xrgwp","openshift-dns/dns-default-5xrdj","openshift-multus/multus-xs8hn","openshift-image-registry/image-registry-59f787ddb6-57tbv","kube-system/konnectivity-agent-d77b4","openshift-insights/insights-runtime-extractor-hvqh6","openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz","openshift-image-registry/node-ca-f2zxw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-fs49h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw","openshift-service-ca/service-ca-5fffc97bdf-sbhs4"]
Oct 30 21:49:49.727594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.727571    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.727818 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.727805    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.733729 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.733692    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:49.733883 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.733692    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.734180 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.734162    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.738831 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.738812    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:49.739074 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.739057    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:49.743252 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.743235    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:49.745549 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.745528    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.746228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.746197    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.747571 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.747554    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.783983 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.783953    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:49:49.786010 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.785991    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:49:49.789359 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.789344    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.790191 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.790177    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.790276 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.790226    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:49:49.790319 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.790277    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:49:49.792817 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.792795    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.793341 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.793324    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.793432 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.793372    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.793489 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.793383    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.793489 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.793474    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:49:49.793914 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.793897    5031 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:49:49.795788 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.795764    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:49.795871 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.795817    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.796288 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796271    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:49:49.796288 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796281    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:49:49.796288 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796293    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:49:49.796449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796322    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:49:49.796449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796326    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:49:49.796449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796342    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:49:49.796449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796350    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:49:49.796449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796356    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:49:49.796449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796364    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:49.796449 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.796405    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal"
Oct 30 21:49:49.796676 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.796489    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:49:49.804744 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804720    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-kubelet-config\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.804821 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804754    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-run\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.804821 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804778    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-systemd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.804901 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804845    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.804901 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804884    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.804971 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804911    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cnibin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.804971 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804935    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-netns\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.804971 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804957    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.805111 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.804979    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268dd18a-b222-4c6d-b981-6babbe21878f-iptables-alerter-script\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:49.805111 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805061    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cni-binary-copy\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.805253 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805137    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-netns\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.805253 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805170    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8js\" (UniqueName: \"kubernetes.io/projected/9aaad021-a27a-4ee3-97d7-71fbaa1aff87-kube-api-access-gb8js\") pod \"kas-connection-checker-685895c7c9-w9w7t\" (UID: \"9aaad021-a27a-4ee3-97d7-71fbaa1aff87\") " pod="kube-system/kas-connection-checker-685895c7c9-w9w7t"
Oct 30 21:49:49.805253 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805203    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8aed736-cadf-4091-b78e-1d555860a055-data-volume\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.805426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805260    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4qp\" (UniqueName: \"kubernetes.io/projected/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-kube-api-access-ns4qp\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:49.805426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805296    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.805426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805315    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8aed736-cadf-4091-b78e-1d555860a055-data-volume\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.805426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805319    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-node-log\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.805426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805393    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.805709 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805518    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.805709 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805572    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-sys\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.805709 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805597    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-host\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.805709 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805621    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-cert\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:49.805845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805703    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkx6\" (UniqueName: \"kubernetes.io/projected/268dd18a-b222-4c6d-b981-6babbe21878f-kube-api-access-ltkx6\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:49.805980 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805961    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-tmp\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.806019 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.805998    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:49.806097 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806082    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-tmp\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.806133 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806116    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.806167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806143    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mrs4\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.806198 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806168    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:49.806312 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806289    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-cabundle\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:49.806415 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806333    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4775f051-dc74-47a3-b528-a735daac1287-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:49.806415 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806404    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqkh\" (UniqueName: \"kubernetes.io/projected/f00a23f9-5c7e-437b-929c-4f793b200fca-kube-api-access-plqkh\") pod \"network-check-source-6f78d59f99-t9z6m\" (UID: \"f00a23f9-5c7e-437b-929c-4f793b200fca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"
Oct 30 21:49:49.806554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806448    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.806554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806459    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-modprobe-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.806554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806488    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4775f051-dc74-47a3-b528-a735daac1287-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:49.806554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806511    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:49.806554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806542    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbl8q\" (UniqueName: \"kubernetes.io/projected/d8aed736-cadf-4091-b78e-1d555860a055-kube-api-access-mbl8q\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.806787 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806596    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-var-lib-kubelet\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.806787 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806644    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2198f51-8630-484c-9825-bb91b3b897e4-config-volume\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.806787 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806673    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.806787 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806737    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-trusted-ca\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.807007 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806801    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.807007 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806888    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/99936bc2-3cbd-445f-9a02-f387cedc1438-kube-api-access-9tww6\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:49.807007 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806912    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-hostroot\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.807007 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806933    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-bin\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.807007 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.806971    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.807287 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807031    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8vz\" (UniqueName: \"kubernetes.io/projected/bcde6ed5-cbad-482d-9f42-b6218e390bd2-kube-api-access-lj8vz\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:49.807287 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807066    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfjj\" (UniqueName: \"kubernetes.io/projected/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-kube-api-access-czfjj\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.807287 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807075    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-trusted-ca\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.807287 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807256    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.807287 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807286    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysconfig\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.807556 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807308    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-kubernetes\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.807556 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807348    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42fq\" (UniqueName: \"kubernetes.io/projected/c2198f51-8630-484c-9825-bb91b3b897e4-kube-api-access-m42fq\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.807556 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807392    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-k8s-cni-cncf-io\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.807556 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807415    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-multus\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.807556 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807467    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwm4\" (UniqueName: \"kubernetes.io/projected/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-kube-api-access-scwm4\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:49.807834 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807582    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.807834 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807626    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnmz\" (UniqueName: \"kubernetes.io/projected/9a549fa1-91e2-4c55-b306-2baf8e54dd96-kube-api-access-4fnmz\") pod \"volume-data-source-validator-58c9f5f46d-nxqdw\" (UID: \"9a549fa1-91e2-4c55-b306-2baf8e54dd96\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"
Oct 30 21:49:49.807834 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807684    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-stats-auth\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.807834 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807733    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.807834 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807831    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzc2\" (UniqueName: \"kubernetes.io/projected/5719f87e-6553-4207-8b40-766d750d8b4c-kube-api-access-jlzc2\") pod \"migrator-66df5d496b-k6mjh\" (UID: \"5719f87e-6553-4207-8b40-766d750d8b4c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"
Oct 30 21:49:49.808157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807879    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-bin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.808157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.807936    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-stats-auth\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.808157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808000    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-log-socket\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.808157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808044    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.808157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808082    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssfj\" (UniqueName: \"kubernetes.io/projected/d07ee097-855f-4598-a947-2268077bac00-kube-api-access-6ssfj\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.808157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808111    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-ovn\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.808157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808137    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hr9\" (UniqueName: \"kubernetes.io/projected/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-kube-api-access-r4hr9\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808184    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808272    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-dbus\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808291    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9k76\" (UniqueName: \"kubernetes.io/projected/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kube-api-access-n9k76\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808315    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808321    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808342    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-serving-cert\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808361    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808378    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4572a1ca-c58c-4207-a080-7dae7263882a-hosts-file\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808402    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808416    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808435    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-var-lib-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808450    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-config\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808465    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htbb\" (UniqueName: \"kubernetes.io/projected/30369506-d839-4e0b-a652-a3875a88b1b4-kube-api-access-9htbb\") pod \"migrator-66df5d496b-66hdp\" (UID: \"30369506-d839-4e0b-a652-a3875a88b1b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"
Oct 30 21:49:49.808516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808501    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59dafadb-fce7-4a75-afc2-fc4275882723-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808653    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07ee097-855f-4598-a947-2268077bac00-config\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808700    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-slash\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808736    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-script-lib\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808752    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808793    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808810    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808824    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-etc-kubernetes\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808831    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808851    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cnibin\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808906    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-multus-certs\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808924    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808940    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ct75\" (UniqueName: \"kubernetes.io/projected/4572a1ca-c58c-4207-a080-7dae7263882a-kube-api-access-5ct75\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808962    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-serving-cert\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.808975    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809001    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4775f051-dc74-47a3-b528-a735daac1287-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809022    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-serving-cert\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809040    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809056    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8aed736-cadf-4091-b78e-1d555860a055-crio-socket\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809092    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809114    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809130    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268dd18a-b222-4c6d-b981-6babbe21878f-host-slash\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809146    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzbf\" (UniqueName: \"kubernetes.io/projected/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-kube-api-access-gfzbf\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809163    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:49.809173 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809182    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809201    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-system-cni-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809234    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809232    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-serving-cert\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809405    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-systemd-units\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809425    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809445    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809459    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809743    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472cs\" (UniqueName: \"kubernetes.io/projected/1fdc58d1-11b9-4879-8879-8909f186bbe5-kube-api-access-472cs\") pod \"kas-connection-checker-685895c7c9-jqx4n\" (UID: \"1fdc58d1-11b9-4879-8879-8909f186bbe5\") " pod="kube-system/kas-connection-checker-685895c7c9-jqx4n"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809800    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sc26\" (UniqueName: \"kubernetes.io/projected/4775f051-dc74-47a3-b528-a735daac1287-kube-api-access-6sc26\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809889    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.809997    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-systemd\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810036    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810071    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8n9\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810103    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-tuned\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.810268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-daemon-config\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.810924 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810647    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-netd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.810924 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810685    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.810924 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810725    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxhv\" (UniqueName: \"kubernetes.io/projected/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-kube-api-access-wfxhv\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:49.810924 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810760    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bcde6ed5-cbad-482d-9f42-b6218e390bd2-serviceca\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:49.810924 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810789    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-key\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:49.810924 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810821    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-default-certificate\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810929    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-etc-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810961    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.810992    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7q47\" (UniqueName: \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811024    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811059    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-system-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811091    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-config\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811118    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811151    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-snapshots\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.811192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811184    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:49.811612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811235    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-conf\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.811612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811265    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-conf-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.811612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811318    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.811612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811521    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.811612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811550    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4572a1ca-c58c-4207-a080-7dae7263882a-tmp-dir\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:49.811612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811582    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2198f51-8630-484c-9825-bb91b3b897e4-tmp-dir\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.811892 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811615    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-kubelet\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.811892 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811646    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-env-overrides\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.811892 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811672    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ae2e290-c982-4b8a-a815-d3babf2679ba-agent-certs\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:49.811892 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811712    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2e290-c982-4b8a-a815-d3babf2679ba-konnectivity-ca\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:49.811892 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.811748    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr2j\" (UniqueName: \"kubernetes.io/projected/920770c6-fd18-41d7-95f6-a55c195361e6-kube-api-access-6cr2j\") pod \"kas-connection-checker-685895c7c9-m87n9\" (UID: \"920770c6-fd18-41d7-95f6-a55c195361e6\") " pod="kube-system/kas-connection-checker-685895c7c9-m87n9"
Oct 30 21:49:49.812194 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812173    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-tuned\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.812288 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812272    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-snapshots\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.812346 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812319    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.812346 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812339    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4572a1ca-c58c-4207-a080-7dae7263882a-tmp-dir\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:49.812436 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812357    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-tmp\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.812436 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812387    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dssr\" (UniqueName: \"kubernetes.io/projected/47bc5387-3f28-4d19-9f51-cdd6e298896f-kube-api-access-6dssr\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.812436 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812397    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2198f51-8630-484c-9825-bb91b3b897e4-tmp-dir\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.812559 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812433    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-socket-dir-parent\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.812559 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812467    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-kubelet\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.812559 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812525    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-config\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:49.812684 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812556    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcde6ed5-cbad-482d-9f42-b6218e390bd2-host\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:49.812684 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812590    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07ee097-855f-4598-a947-2268077bac00-serving-cert\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.812684 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812625    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.812684 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812659    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2q97\" (UniqueName: \"kubernetes.io/projected/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-kube-api-access-s2q97\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.812855 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812685    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.812855 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812809    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-default-certificate\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.812855 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812823    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdxm\" (UniqueName: \"kubernetes.io/projected/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-kube-api-access-nkdxm\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:49.812988 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812856    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.812988 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812888    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmmt\" (UniqueName: \"kubernetes.io/projected/bf193224-26cc-4cd6-8aa5-2424e1212c2a-kube-api-access-7cmmt\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:49.812988 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812920    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-lib-modules\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.812988 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812952    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcbg\" (UniqueName: \"kubernetes.io/projected/5a6789d1-ca4c-4efe-b973-8c764ae03b89-kube-api-access-tlcbg\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.812988 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.812985    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.813230 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.813016    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf193224-26cc-4cd6-8aa5-2424e1212c2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:49.813230 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.813054    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.813230 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.813131    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-tmp\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.813374 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.813346    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-config\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:49.813771 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.813751    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07ee097-855f-4598-a947-2268077bac00-serving-cert\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.835983 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.835908    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs82w\""
Oct 30 21:49:49.835983 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.835946    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nvz\""
Oct 30 21:49:49.836235 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.836188    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:49:49.836364 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.836309    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhgs\""
Oct 30 21:49:49.836557 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.836545    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mdmg\""
Oct 30 21:49:49.836617 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.836605    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jjx9q\""
Oct 30 21:49:49.836686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.836664    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k97q2\""
Oct 30 21:49:49.838585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.838549    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:49:49.838994 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.838887    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vjtf6\""
Oct 30 21:49:49.838994 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.838938    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gmxqg\""
Oct 30 21:49:49.838994 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.838978    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:49:49.839185 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839167    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:49:49.839315 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839299    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pqcb5\""
Oct 30 21:49:49.839373 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839333    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lwnsl\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839483    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bhhxg\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839491    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839535    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839551    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qmkfs\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839587    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-84vh4\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839635    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839642    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rrcm9\""
Oct 30 21:49:49.839682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.839663    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wvpps\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840155    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840245    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5s29\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840269    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-td54n\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840327    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c6mrn\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840351    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840364    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9pjm\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840405    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840410    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:49:49.840488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.840448    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-original-pull-secret\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.841952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.841927    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jm7gq\""
Oct 30 21:49:49.842024 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.841995    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-key\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:49.842862 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.842843    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:49.843857 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.843843    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ae2e290-c982-4b8a-a815-d3babf2679ba-agent-certs\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:49.845926 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.845912    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.845995 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.845927    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-cert\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:49.869642 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.869609    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8js\" (UniqueName: \"kubernetes.io/projected/9aaad021-a27a-4ee3-97d7-71fbaa1aff87-kube-api-access-gb8js\") pod \"kas-connection-checker-685895c7c9-w9w7t\" (UID: \"9aaad021-a27a-4ee3-97d7-71fbaa1aff87\") " pod="kube-system/kas-connection-checker-685895c7c9-w9w7t"
Oct 30 21:49:49.876265 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.876243    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.889021 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.888995    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.890886 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.890863    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.890989 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.890889    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:49:49.891118 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891099    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:49:49.891311 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891185    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.891311 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891195    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.891311 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891230    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.891311 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891255    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:49:49.891311 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891262    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:49:49.891554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891343    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.891554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891364    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.891554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891364    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.891554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891410    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.891554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891552    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:49:49.891782 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891606    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:49:49.891782 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891611    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.891782 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891661    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.891953 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891940    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:49:49.892001 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891955    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.892001 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891992    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:49:49.892092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891999    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892010    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892034    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.891998    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.892092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892051    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:49:49.892092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892051    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892077    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.892410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892154    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892225    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:49:49.892410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892257    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:49:49.892410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892277    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892358    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.892410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892359    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.892410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892371    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:49:49.892697 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892436    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:49:49.892697 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892447    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892697 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892547    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.892697 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.892547    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893053    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893101    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893196    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893249    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-env-overrides\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893261    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893297    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893302    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893315    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893339    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893360    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893377    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893377    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893341    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.893438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893410    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:49:49.894095 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893590    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42fq\" (UniqueName: \"kubernetes.io/projected/c2198f51-8630-484c-9825-bb91b3b897e4-kube-api-access-m42fq\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.894095 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.893914    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.894095 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.894044    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:49:49.894095 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.894053    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:49:49.894292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.894188    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:49:49.895157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.895140    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.895686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.895304    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.895686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.895496    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzc2\" (UniqueName: \"kubernetes.io/projected/5719f87e-6553-4207-8b40-766d750d8b4c-kube-api-access-jlzc2\") pod \"migrator-66df5d496b-k6mjh\" (UID: \"5719f87e-6553-4207-8b40-766d750d8b4c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh"
Oct 30 21:49:49.895686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.895570    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268dd18a-b222-4c6d-b981-6babbe21878f-iptables-alerter-script\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:49.895686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.895614    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbl8q\" (UniqueName: \"kubernetes.io/projected/d8aed736-cadf-4091-b78e-1d555860a055-kube-api-access-mbl8q\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.895686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.895645    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cni-binary-copy\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.895686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.895655    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mrs4\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.896539 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.896522    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:49:49.896681 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.896660    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-signing-cabundle\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:49.896752 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.896731    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.896805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.896772    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:49.896947 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.896925    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2198f51-8630-484c-9825-bb91b3b897e4-config-volume\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.897128 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.897113    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqkh\" (UniqueName: \"kubernetes.io/projected/f00a23f9-5c7e-437b-929c-4f793b200fca-kube-api-access-plqkh\") pod \"network-check-source-6f78d59f99-t9z6m\" (UID: \"f00a23f9-5c7e-437b-929c-4f793b200fca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9z6m"
Oct 30 21:49:49.897181 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.897155    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899016    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899066    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59dafadb-fce7-4a75-afc2-fc4275882723-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899358    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899397    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8vz\" (UniqueName: \"kubernetes.io/projected/bcde6ed5-cbad-482d-9f42-b6218e390bd2-kube-api-access-lj8vz\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899466    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkx6\" (UniqueName: \"kubernetes.io/projected/268dd18a-b222-4c6d-b981-6babbe21878f-kube-api-access-ltkx6\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899554    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4775f051-dc74-47a3-b528-a735daac1287-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899662    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-script-lib\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.899724 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899663    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwm4\" (UniqueName: \"kubernetes.io/projected/eaec4c99-a4dd-47e4-9f3d-f6a276f166a0-kube-api-access-scwm4\") pod \"ingress-canary-xrgwp\" (UID: \"eaec4c99-a4dd-47e4-9f3d-f6a276f166a0\") " pod="openshift-ingress-canary/ingress-canary-xrgwp"
Oct 30 21:49:49.900150 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899806    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.900150 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.899810    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.900150 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.900111    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.900356 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.900337    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4qp\" (UniqueName: \"kubernetes.io/projected/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-kube-api-access-ns4qp\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:49.900403 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.900387    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-daemon-config\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.900538 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.900524    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfjj\" (UniqueName: \"kubernetes.io/projected/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-kube-api-access-czfjj\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.901311 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.901289    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/99936bc2-3cbd-445f-9a02-f387cedc1438-kube-api-access-9tww6\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:49.901416 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.901374    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnmz\" (UniqueName: \"kubernetes.io/projected/9a549fa1-91e2-4c55-b306-2baf8e54dd96-kube-api-access-4fnmz\") pod \"volume-data-source-validator-58c9f5f46d-nxqdw\" (UID: \"9a549fa1-91e2-4c55-b306-2baf8e54dd96\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nxqdw"
Oct 30 21:49:49.902139 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.902122    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bcde6ed5-cbad-482d-9f42-b6218e390bd2-serviceca\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:49.902197 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.902141    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.902197 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.902150    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-ovnkube-config\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.902334 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.902315    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2e290-c982-4b8a-a815-d3babf2679ba-konnectivity-ca\") pod \"konnectivity-agent-d77b4\" (UID: \"7ae2e290-c982-4b8a-a815-d3babf2679ba\") " pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:49:49.903520 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.903504    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf193224-26cc-4cd6-8aa5-2424e1212c2a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:49.905968 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.905951    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htbb\" (UniqueName: \"kubernetes.io/projected/30369506-d839-4e0b-a652-a3875a88b1b4-kube-api-access-9htbb\") pod \"migrator-66df5d496b-66hdp\" (UID: \"30369506-d839-4e0b-a652-a3875a88b1b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp"
Oct 30 21:49:49.906046 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.906000    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcbg\" (UniqueName: \"kubernetes.io/projected/5a6789d1-ca4c-4efe-b973-8c764ae03b89-kube-api-access-tlcbg\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.906439 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.906421    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmmt\" (UniqueName: \"kubernetes.io/projected/bf193224-26cc-4cd6-8aa5-2424e1212c2a-kube-api-access-7cmmt\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:49.913455 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913438    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.913545 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913463    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.913545 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913479    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-lib-modules\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.913545 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913496    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-kubelet-config\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.913545 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913519    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-run\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.913545 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913541    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-systemd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913564    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cnibin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913570    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913587    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-netns\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913589    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913613    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-netns\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913624    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-run\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913626    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-lib-modules\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913645    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913654    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-cnibin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913663    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-kubelet-config\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913677    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-systemd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913689    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-node-log\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913701    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-netns\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913709    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913713    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-netns\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913714    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913739    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-node-log\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913767    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-sys\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913789    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-host\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913819    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913827    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-sys\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.913830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913828    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-host\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.913791    5031 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913847    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913881    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-modprobe-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913906    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-var-lib-kubelet\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.913931    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.413911443 +0000 UTC m=+1.938054198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913946    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-var-lib-kubelet\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.913961    5031 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914011    5031 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914006    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.413994603 +0000 UTC m=+1.938137347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914043    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.414033602 +0000 UTC m=+1.938176386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914053    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-modprobe-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.913960    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914082    5031 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914107    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.414099742 +0000 UTC m=+1.938242483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914105    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-hostroot\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914127    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-bin\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914153    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914136    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-hostroot\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914175    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-bin\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914185    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysconfig\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914230    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-kubernetes\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914235    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914257    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-k8s-cni-cncf-io\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914272    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysconfig\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914281    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-multus\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914291    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-kubernetes\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914302    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-k8s-cni-cncf-io\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914311    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914319    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-multus\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914356    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914419    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-bin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914446    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-log-socket\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914471    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-ovn\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914497    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-dbus\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914502    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-log-socket\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914544    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-run-ovn\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914522    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4572a1ca-c58c-4207-a080-7dae7263882a-hosts-file\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914471    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-cni-bin\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914590    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.41457345 +0000 UTC m=+1.938716195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914607    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914618    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5-dbus\") pod \"global-pull-secret-syncer-jl8q2\" (UID: \"2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5\") " pod="kube-system/global-pull-secret-syncer-jl8q2"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914631    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914654    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-var-lib-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914668    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914683    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-slash\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914690    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8aed736-cadf-4091-b78e-1d555860a055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914711    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914714    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4572a1ca-c58c-4207-a080-7dae7263882a-hosts-file\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914753    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-var-lib-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914779    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-d\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914780    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-etc-kubernetes\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914790    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914801    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914807    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-slash\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cnibin\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914833    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-etc-kubernetes\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914834    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-multus-certs\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914856    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-cnibin\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.914863    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.414846527 +0000 UTC m=+1.938989277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:49.914845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914875    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-run-multus-certs\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914901    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8aed736-cadf-4091-b78e-1d555860a055-crio-socket\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914927    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914954    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8aed736-cadf-4091-b78e-1d555860a055-crio-socket\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914954    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914987    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268dd18a-b222-4c6d-b981-6babbe21878f-host-slash\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.914996    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915008    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-system-cni-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915023    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915037    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-systemd-units\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915041    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915053    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915072    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-systemd\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915080    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-system-cni-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915088    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-netd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-cni-netd\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915118    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-etc-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915138    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915145    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-etc-openvswitch\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915156    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915172    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-system-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915187    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915188    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268dd18a-b222-4c6d-b981-6babbe21878f-host-slash\") pod \"iptables-alerter-29pdk\" (UID: \"268dd18a-b222-4c6d-b981-6babbe21878f\") " pod="openshift-network-operator/iptables-alerter-29pdk"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915246    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwxtj\" (UID: \"56d6e046-0ec6-4b8a-bf36-c43a3abd5ba4\") " pod="openshift-multus/multus-additional-cni-plugins-dwxtj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915247    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915278    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915277    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-conf\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915300    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-conf-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915314    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915330    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-kubelet\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915349    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-socket-dir-parent\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915363    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-kubelet\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915377    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcde6ed5-cbad-482d-9f42-b6218e390bd2-host\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915386    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-sysctl-conf\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915394    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915439    5031 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915466    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.415456952 +0000 UTC m=+1.939599693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : secret "router-metrics-certs-default" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915487    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-systemd-units\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915509    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915525    5031 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915539    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6789d1-ca4c-4efe-b973-8c764ae03b89-etc-systemd\") pod \"tuned-rcxbn\" (UID: \"5a6789d1-ca4c-4efe-b973-8c764ae03b89\") " pod="openshift-cluster-node-tuning-operator/tuned-rcxbn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915568    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.415554785 +0000 UTC m=+1.939697547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915579    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915586    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915605    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.415598947 +0000 UTC m=+1.939741689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915619    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915627    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-host-var-lib-kubelet\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915657    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-socket-dir-parent\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915698    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-system-cni-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915700    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcde6ed5-cbad-482d-9f42-b6218e390bd2-host\") pod \"node-ca-f2zxw\" (UID: \"bcde6ed5-cbad-482d-9f42-b6218e390bd2\") " pod="openshift-image-registry/node-ca-f2zxw"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915718    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-multus-conf-dir\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915757    5031 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:49.915791    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:50.415779721 +0000 UTC m=+1.939922485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : secret "metrics-daemon-secret" not found
Oct 30 21:49:49.916897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.915812    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-host-kubelet\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.941236 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.941100    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-472cs\" (UniqueName: \"kubernetes.io/projected/1fdc58d1-11b9-4879-8879-8909f186bbe5-kube-api-access-472cs\") pod \"kas-connection-checker-685895c7c9-jqx4n\" (UID: \"1fdc58d1-11b9-4879-8879-8909f186bbe5\") " pod="kube-system/kas-connection-checker-685895c7c9-jqx4n"
Oct 30 21:49:49.941236 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.941119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxhv\" (UniqueName: \"kubernetes.io/projected/a7c89d90-bbaa-4f37-acb6-c83e561a1d72-kube-api-access-wfxhv\") pod \"service-ca-operator-544d749c8c-9994s\" (UID: \"a7c89d90-bbaa-4f37-acb6-c83e561a1d72\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s"
Oct 30 21:49:49.941738 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.941675    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7q47\" (UniqueName: \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\") pod \"periodic-gathering-scxcw-zq8dc\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") " pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:49:49.941738 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.941683    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8n9\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:49.941903 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.941747    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2q97\" (UniqueName: \"kubernetes.io/projected/e84861d0-c9a6-480b-82ed-5a0ce96dab7d-kube-api-access-s2q97\") pod \"insights-operator-7c4986d449-xh5b2\" (UID: \"e84861d0-c9a6-480b-82ed-5a0ce96dab7d\") " pod="openshift-insights/insights-operator-7c4986d449-xh5b2"
Oct 30 21:49:49.942576 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.942250    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hr9\" (UniqueName: \"kubernetes.io/projected/45bbb59a-b1e7-4993-8a0b-b82f54b1c844-kube-api-access-r4hr9\") pod \"ovnkube-node-8h6kj\" (UID: \"45bbb59a-b1e7-4993-8a0b-b82f54b1c844\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:49.942576 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.942283    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ct75\" (UniqueName: \"kubernetes.io/projected/4572a1ca-c58c-4207-a080-7dae7263882a-kube-api-access-5ct75\") pod \"node-resolver-q5tqg\" (UID: \"4572a1ca-c58c-4207-a080-7dae7263882a\") " pod="openshift-dns/node-resolver-q5tqg"
Oct 30 21:49:49.942576 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.942312    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9k76\" (UniqueName: \"kubernetes.io/projected/7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1-kube-api-access-n9k76\") pod \"aws-ebs-csi-driver-node-b6gws\" (UID: \"7b8cd4cd-5b2d-400f-b3b0-f84abc42bcf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6gws"
Oct 30 21:49:49.942576 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.942444    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr2j\" (UniqueName: \"kubernetes.io/projected/920770c6-fd18-41d7-95f6-a55c195361e6-kube-api-access-6cr2j\") pod \"kas-connection-checker-685895c7c9-m87n9\" (UID: \"920770c6-fd18-41d7-95f6-a55c195361e6\") " pod="kube-system/kas-connection-checker-685895c7c9-m87n9"
Oct 30 21:49:49.942802 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.942649    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzbf\" (UniqueName: \"kubernetes.io/projected/fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82-kube-api-access-gfzbf\") pod \"multus-xs8hn\" (UID: \"fdf66beb-ad8b-47f2-89cc-9b8e6bc18b82\") " pod="openshift-multus/multus-xs8hn"
Oct 30 21:49:49.942802 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.942691    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:49.942903 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.942891    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdxm\" (UniqueName: \"kubernetes.io/projected/9cd08e0e-c491-487a-a4f6-a0955b41c4a0-kube-api-access-nkdxm\") pod \"service-ca-5fffc97bdf-sbhs4\" (UID: \"9cd08e0e-c491-487a-a4f6-a0955b41c4a0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sbhs4"
Oct 30 21:49:49.944940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.943311    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sc26\" (UniqueName: \"kubernetes.io/projected/4775f051-dc74-47a3-b528-a735daac1287-kube-api-access-6sc26\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bz59r\" (UID: \"4775f051-dc74-47a3-b528-a735daac1287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r"
Oct 30 21:49:49.944940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.943766    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6rv\" (UniqueName: \"kubernetes.io/projected/e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66-kube-api-access-pc6rv\") pod \"network-check-target-cjd7c\" (UID: \"e9a7ec18-5d7b-4dc1-89f1-58effdc9ba66\") " pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:49.944940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.943985    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssfj\" (UniqueName: \"kubernetes.io/projected/d07ee097-855f-4598-a947-2268077bac00-kube-api-access-6ssfj\") pod \"console-operator-7cd56b8dbc-fs49h\" (UID: \"d07ee097-855f-4598-a947-2268077bac00\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:49.944940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:49.944343    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dssr\" (UniqueName: \"kubernetes.io/projected/47bc5387-3f28-4d19-9f51-cdd6e298896f-kube-api-access-6dssr\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:50.029630 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.029599    5031 scope.go:122] "RemoveContainer" containerID="0124511bc3d9d40471df983f4e5b72596bc0e86b208962998ff187d6d35cf3be"
Oct 30 21:49:50.050246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.050225    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:49:50.081164 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.081126    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:50.321500 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.321470    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:50.362058 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.361994    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f2zxw_bcde6ed5-cbad-482d-9f42-b6218e390bd2/node-ca/0.log"
Oct 30 21:49:50.419845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.419814    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:50.420009 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.419866    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:50.420009 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.419915    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:50.420009 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.419943    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:50.420009 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.419966    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.419943809 +0000 UTC m=+2.944086568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.420014    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420026    5031 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420028    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420047    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.420062    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420070    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420056314 +0000 UTC m=+2.944199068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : secret "metrics-daemon-secret" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420026    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420082    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420096    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420087063 +0000 UTC m=+2.944229808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420111    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420103391 +0000 UTC m=+2.944246134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420125    5031 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.420152    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420169    5031 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420177    5031 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.420182    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:50.420195 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420204    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420196151 +0000 UTC m=+2.944338917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420258    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420241331 +0000 UTC m=+2.944384092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420264    5031 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.420284    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420310    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420298962 +0000 UTC m=+2.944441742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420331    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420323026 +0000 UTC m=+2.944465793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : secret "router-metrics-certs-default" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420344    5031 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.420359    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420384    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420372274 +0000 UTC m=+2.944515015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420437    5031 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:50.420771 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:50.420470    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:51.420461365 +0000 UTC m=+2.944604130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:50.574070 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.574037    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:50.734533 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.734449    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:49:50.734682 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.734573    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h" event={"ID":"d07ee097-855f-4598-a947-2268077bac00","Type":"ContainerStarted","Data":"fe134a53d1555b1ffaa815b3bddc41a50dc510e477cb1edffad42964f7b44dd1"}
Oct 30 21:49:50.836302 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.836273    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:50.937511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:50.937485    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xrgwp_eaec4c99-a4dd-47e4-9f3d-f6a276f166a0/serve-healthcheck-canary/0.log"
Oct 30 21:49:51.430478 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430448    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:51.430669 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430497    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:51.430669 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430540    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:51.430669 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430565    5031 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:51.430669 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430629    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.430614975 +0000 UTC m=+4.954757736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:51.430669 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430571    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430683    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430642    5031 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430721    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430758    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.430744058 +0000 UTC m=+4.954886805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430641    5031 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430786    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.430777172 +0000 UTC m=+4.954919945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430796    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430658    5031 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430807    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430807    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.430796362 +0000 UTC m=+4.954939126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:51.430873 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430852    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430880    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430908    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430945    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430957    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.430942348 +0000 UTC m=+4.955085092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.430946    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430978    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.430968684 +0000 UTC m=+4.955111435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430959    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.431023    5031 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.430997    5031 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.431051    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.431039911 +0000 UTC m=+4.955182654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.431111    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs podName:99936bc2-3cbd-445f-9a02-f387cedc1438 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.431099944 +0000 UTC m=+4.955242686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs") pod "network-metrics-daemon-h8h85" (UID: "99936bc2-3cbd-445f-9a02-f387cedc1438") : secret "metrics-daemon-secret" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:51.431126    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert podName:59dafadb-fce7-4a75-afc2-fc4275882723 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:53.431118623 +0000 UTC m=+4.955261365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g64qk" (UID: "59dafadb-fce7-4a75-afc2-fc4275882723") : secret "networking-console-plugin-cert" not found
Oct 30 21:49:51.431306 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.431269    5031 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:49:51.433522 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.433507    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47bc5387-3f28-4d19-9f51-cdd6e298896f-metrics-certs\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:51.456602 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.456531    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:51.537405 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.537326    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-bz59r_4775f051-dc74-47a3-b528-a735daac1287/kube-storage-version-migrator-operator/0.log"
Oct 30 21:49:51.685006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.684971    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:51.739637 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.739605    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66hdp" event={"ID":"30369506-d839-4e0b-a652-a3875a88b1b4","Type":"ContainerStarted","Data":"ec4897334785f0a77c42abcaddf4631012e9ea08c15ac1f086c56068c4eea445"}
Oct 30 21:49:51.741082 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.741049    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6mjh" event={"ID":"5719f87e-6553-4207-8b40-766d750d8b4c","Type":"ContainerStarted","Data":"4a0aefae18c42181e1a95bf2b012bf8663188b37317bcbbc090b088a66d50d79"}
Oct 30 21:49:51.951492 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:51.951455    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:52.186604 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:52.186573    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:52.487643 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:52.487613    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:52.730533 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:52.730491    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:52.958897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:52.958871    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:53.130091 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.130059    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:49:53.450081 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450041    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450093    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450112    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450132    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450177    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450195    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450225    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450238    5031 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450244    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698788495-986ld: secret "image-registry-tls" not found
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450257    5031 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59f787ddb6-57tbv: secret "image-registry-tls" not found
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450242    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450305    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls podName:2e458ae3-5f11-4650-ae5e-0145b063be66 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:57.450288196 +0000 UTC m=+8.974430965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls") pod "image-registry-8698788495-986ld" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66") : secret "image-registry-tls" not found
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450307    5031 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:53.450316 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450320    5031 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450352    5031 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450363    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls podName:3faa0473-4e54-4ef1-adf1-17f86a91bdd7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:57.450347687 +0000 UTC m=+8.974490430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls") pod "image-registry-59f787ddb6-57tbv" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7") : secret "image-registry-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450401    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls podName:bf193224-26cc-4cd6-8aa5-2424e1212c2a nodeName:}" failed. No retries permitted until 2026-10-30 21:49:57.450379533 +0000 UTC m=+8.974522275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6spwz" (UID: "bf193224-26cc-4cd6-8aa5-2424e1212c2a") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450427    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:57.450417476 +0000 UTC m=+8.974560219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450448    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:57.450439753 +0000 UTC m=+8.974582494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450493    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.450537    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450647    5031 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450690    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls podName:c2198f51-8630-484c-9825-bb91b3b897e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:57.450673976 +0000 UTC m=+8.974816731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls") pod "dns-default-5xrdj" (UID: "c2198f51-8630-484c-9825-bb91b3b897e4") : secret "dns-default-metrics-tls" not found
Oct 30 21:49:53.450805 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:53.450712    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle podName:47bc5387-3f28-4d19-9f51-cdd6e298896f nodeName:}" failed. No retries permitted until 2026-10-30 21:49:57.450701283 +0000 UTC m=+8.974844039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle") pod "router-default-d48b6d977-dstqd" (UID: "47bc5387-3f28-4d19-9f51-cdd6e298896f") : configmap references non-existent config key: service-ca.crt
Oct 30 21:49:53.452740 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.452712    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99936bc2-3cbd-445f-9a02-f387cedc1438-metrics-certs\") pod \"network-metrics-daemon-h8h85\" (UID: \"99936bc2-3cbd-445f-9a02-f387cedc1438\") " pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:53.452825 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.452785    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59dafadb-fce7-4a75-afc2-fc4275882723-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g64qk\" (UID: \"59dafadb-fce7-4a75-afc2-fc4275882723\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:53.640200 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.640165    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk"
Oct 30 21:49:53.644300 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.644278    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8h85"
Oct 30 21:49:53.647635 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:53.647612    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:54.028948 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.028872    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-g64qk"]
Oct 30 21:49:54.031979 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:54.031950    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dafadb_fce7_4a75_afc2_fc4275882723.slice/crio-bbc02910230d2392e18a641fd5d77634905217aab457136c69e49fdb62665edb WatchSource:0}: Error finding container bbc02910230d2392e18a641fd5d77634905217aab457136c69e49fdb62665edb: Status 404 returned error can't find the container with id bbc02910230d2392e18a641fd5d77634905217aab457136c69e49fdb62665edb
Oct 30 21:49:54.040705 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.040683    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:54.079328 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.079294    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h8h85"]
Oct 30 21:49:54.081656 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:54.081627    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99936bc2_3cbd_445f_9a02_f387cedc1438.slice/crio-adccfa5e944dcf835d712ee3e20015fa65829b390aeeb0d7a530b99ea153be3b WatchSource:0}: Error finding container adccfa5e944dcf835d712ee3e20015fa65829b390aeeb0d7a530b99ea153be3b: Status 404 returned error can't find the container with id adccfa5e944dcf835d712ee3e20015fa65829b390aeeb0d7a530b99ea153be3b
Oct 30 21:49:54.267529 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.267494    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:54.490472 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.490440    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:54.754001 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.753922    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk" event={"ID":"59dafadb-fce7-4a75-afc2-fc4275882723","Type":"ContainerStarted","Data":"bbc02910230d2392e18a641fd5d77634905217aab457136c69e49fdb62665edb"}
Oct 30 21:49:54.755192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.755170    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8h85" event={"ID":"99936bc2-3cbd-445f-9a02-f387cedc1438","Type":"ContainerStarted","Data":"adccfa5e944dcf835d712ee3e20015fa65829b390aeeb0d7a530b99ea153be3b"}
Oct 30 21:49:54.843526 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:54.843490    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:55.149874 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:55.149808    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:55.471796 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:55.471767    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:55.703722 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:55.703690    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:55.760755 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:55.760666    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8h85" event={"ID":"99936bc2-3cbd-445f-9a02-f387cedc1438","Type":"ContainerStarted","Data":"cd9c7579ca2dedb43f2e9c6ebbb32b05def282b526a63c47b32b1486cb5699fd"}
Oct 30 21:49:55.760933 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:55.760769    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8h85" event={"ID":"99936bc2-3cbd-445f-9a02-f387cedc1438","Type":"ContainerStarted","Data":"2edd90af5de59682065fe362997336679f08108f0a8a4fff13eeba5c39ce96bd"}
Oct 30 21:49:55.762124 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:55.762099    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-g64qk" event={"ID":"59dafadb-fce7-4a75-afc2-fc4275882723","Type":"ContainerStarted","Data":"e464d8a40cf0cd8fc5bb76882aea6e0e11db86f46d4ca2781447f04a52e49c81"}
Oct 30 21:49:56.051616 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:56.051540    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:56.268199 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:56.268168    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:56.489879 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:56.489849    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:56.849756 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:56.849724    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:57.068883 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.068852    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:57.488508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.488466    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.488520    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.488565    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.488611    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.488667    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.488695    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.488719    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:57.488837    5031 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:57.488941 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:57.488913    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls podName:d8aed736-cadf-4091-b78e-1d555860a055 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:05.48889404 +0000 UTC m=+17.013036794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvqh6" (UID: "d8aed736-cadf-4091-b78e-1d555860a055") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:49:57.489345 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:57.489035    5031 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:49:57.489345 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:49:57.489148    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls podName:8f32ae7a-4fbd-4959-a49b-76e9729a5c8e nodeName:}" failed. No retries permitted until 2026-10-30 21:50:05.489111914 +0000 UTC m=+17.013254670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x9wjc" (UID: "8f32ae7a-4fbd-4959-a49b-76e9729a5c8e") : secret "samples-operator-tls" not found
Oct 30 21:49:57.489345 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.489276    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bc5387-3f28-4d19-9f51-cdd6e298896f-service-ca-bundle\") pod \"router-default-d48b6d977-dstqd\" (UID: \"47bc5387-3f28-4d19-9f51-cdd6e298896f\") " pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:57.491850 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.491827    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"image-registry-59f787ddb6-57tbv\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") " pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:57.491956 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.491848    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2198f51-8630-484c-9825-bb91b3b897e4-metrics-tls\") pod \"dns-default-5xrdj\" (UID: \"c2198f51-8630-484c-9825-bb91b3b897e4\") " pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:57.491956 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.491830    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"image-registry-8698788495-986ld\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") " pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:57.491956 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.491908    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf193224-26cc-4cd6-8aa5-2424e1212c2a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6spwz\" (UID: \"bf193224-26cc-4cd6-8aa5-2424e1212c2a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:57.534834 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.534796    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:57.534978 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.534889    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:49:57.534978 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.534921    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"
Oct 30 21:49:57.546292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.546278    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:49:57.546805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.546514    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:49:57.646465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.646438    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:57.877887 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.877853    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:57.960939 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:57.960906    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz"]
Oct 30 21:49:57.964031 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:57.964009    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf193224_26cc_4cd6_8aa5_2424e1212c2a.slice/crio-101781b6e1d93cbd11ac8bc42a7bf67035428090e724bace92eede4506a811de WatchSource:0}: Error finding container 101781b6e1d93cbd11ac8bc42a7bf67035428090e724bace92eede4506a811de: Status 404 returned error can't find the container with id 101781b6e1d93cbd11ac8bc42a7bf67035428090e724bace92eede4506a811de
Oct 30 21:49:58.007536 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:58.007506    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faa0473_4e54_4ef1_adf1_17f86a91bdd7.slice/crio-a14a2884ee18d61d6ff5e674afc1b87690be6aad64fd58a96c9fd511dadc61a8 WatchSource:0}: Error finding container a14a2884ee18d61d6ff5e674afc1b87690be6aad64fd58a96c9fd511dadc61a8: Status 404 returned error can't find the container with id a14a2884ee18d61d6ff5e674afc1b87690be6aad64fd58a96c9fd511dadc61a8
Oct 30 21:49:58.010870 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.010842    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f787ddb6-57tbv"]
Oct 30 21:49:58.021470 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.021451    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5xrdj"]
Oct 30 21:49:58.023963 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:58.023938    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2198f51_8630_484c_9825_bb91b3b897e4.slice/crio-55f07d07fed81bc2c2cda5431bad1223e2f89aa2efb63192ec99cbbe6a1084d6 WatchSource:0}: Error finding container 55f07d07fed81bc2c2cda5431bad1223e2f89aa2efb63192ec99cbbe6a1084d6: Status 404 returned error can't find the container with id 55f07d07fed81bc2c2cda5431bad1223e2f89aa2efb63192ec99cbbe6a1084d6
Oct 30 21:49:58.028159 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.028134    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d48b6d977-dstqd"]
Oct 30 21:49:58.028583 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:58.028560    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bc5387_3f28_4d19_9f51_cdd6e298896f.slice/crio-d6deff438417ff6d868c05a9d63a9366b0c88be64fe5eb2eb0e64014e896ee85 WatchSource:0}: Error finding container d6deff438417ff6d868c05a9d63a9366b0c88be64fe5eb2eb0e64014e896ee85: Status 404 returned error can't find the container with id d6deff438417ff6d868c05a9d63a9366b0c88be64fe5eb2eb0e64014e896ee85
Oct 30 21:49:58.057168 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:49:58.057140    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e458ae3_5f11_4650_ae5e_0145b063be66.slice/crio-6c6ff2cd9a00606e5667af426db29ced97fde5c724ef86bb954d735cc3c7ec2d WatchSource:0}: Error finding container 6c6ff2cd9a00606e5667af426db29ced97fde5c724ef86bb954d735cc3c7ec2d: Status 404 returned error can't find the container with id 6c6ff2cd9a00606e5667af426db29ced97fde5c724ef86bb954d735cc3c7ec2d
Oct 30 21:49:58.061729 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.061709    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8698788495-986ld"]
Oct 30 21:49:58.118053 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.118027    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:58.379694 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.379656    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:58.649360 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.649282    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:58.773063 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.773024    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xrdj" event={"ID":"c2198f51-8630-484c-9825-bb91b3b897e4","Type":"ContainerStarted","Data":"55f07d07fed81bc2c2cda5431bad1223e2f89aa2efb63192ec99cbbe6a1084d6"}
Oct 30 21:49:58.774681 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.774647    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d48b6d977-dstqd" event={"ID":"47bc5387-3f28-4d19-9f51-cdd6e298896f","Type":"ContainerStarted","Data":"fbc10b6e2fd1c97cf4c871f182e81305978d3edd27897463c832d47cc7e11b3c"}
Oct 30 21:49:58.775314 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.774707    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d48b6d977-dstqd" event={"ID":"47bc5387-3f28-4d19-9f51-cdd6e298896f","Type":"ContainerStarted","Data":"d6deff438417ff6d868c05a9d63a9366b0c88be64fe5eb2eb0e64014e896ee85"}
Oct 30 21:49:58.775953 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.775930    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz" event={"ID":"bf193224-26cc-4cd6-8aa5-2424e1212c2a","Type":"ContainerStarted","Data":"101781b6e1d93cbd11ac8bc42a7bf67035428090e724bace92eede4506a811de"}
Oct 30 21:49:58.777346 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.777323    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv" event={"ID":"3faa0473-4e54-4ef1-adf1-17f86a91bdd7","Type":"ContainerStarted","Data":"9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff"}
Oct 30 21:49:58.777448 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.777371    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv" event={"ID":"3faa0473-4e54-4ef1-adf1-17f86a91bdd7","Type":"ContainerStarted","Data":"a14a2884ee18d61d6ff5e674afc1b87690be6aad64fd58a96c9fd511dadc61a8"}
Oct 30 21:49:58.778662 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.778643    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698788495-986ld" event={"ID":"2e458ae3-5f11-4650-ae5e-0145b063be66","Type":"ContainerStarted","Data":"345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb"}
Oct 30 21:49:58.778756 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.778675    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698788495-986ld" event={"ID":"2e458ae3-5f11-4650-ae5e-0145b063be66","Type":"ContainerStarted","Data":"6c6ff2cd9a00606e5667af426db29ced97fde5c724ef86bb954d735cc3c7ec2d"}
Oct 30 21:49:58.915056 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:58.914976    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:59.535340 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.535266    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:59.538285 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.538258    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:49:59.782425 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.782397    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:59.782991 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.782974    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:59.836462 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.836379    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:49:59.836625 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.836607    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:59.842796 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.842767    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:59.852386 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.852358    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:49:59.852572 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.852545    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:59.857018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:49:59.856991    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fs49h"
Oct 30 21:50:00.067640 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.067607    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:00.303013 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.302990    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:00.578909 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.578877    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:00.787843 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.787775    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xrdj" event={"ID":"c2198f51-8630-484c-9825-bb91b3b897e4","Type":"ContainerStarted","Data":"607f850a277a815205765c2cf583e1858d57d4403ecf81b947a39a9f4bebaec9"}
Oct 30 21:50:00.787843 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.787823    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xrdj" event={"ID":"c2198f51-8630-484c-9825-bb91b3b897e4","Type":"ContainerStarted","Data":"4a41b56b0b9757c9c89e12370e67153eba23a6021cf3db393d27e8cfdbc7117a"}
Oct 30 21:50:00.788317 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.787861    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:50:00.789441 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.789419    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6spwz" event={"ID":"bf193224-26cc-4cd6-8aa5-2424e1212c2a","Type":"ContainerStarted","Data":"b3fffc41f90b008da99a9e2c3d9c5325708ece767173ab2a9e09bbd7c37e5db6"}
Oct 30 21:50:00.789608 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.789592    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:50:00.822109 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:00.822081    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:01.150014 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:01.149984    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:01.381991 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:01.381959    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:01.643852 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:01.643815    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:01.873261 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:01.873193    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:02.107577 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:02.107549    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:02.480101 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:02.480001    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:02.715962 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:02.715931    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:03.666735 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:03.666703    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:50:03.667178 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:03.666824    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:50:03.667986 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:03.667967    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d48b6d977-dstqd"
Oct 30 21:50:04.070750 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:04.070672    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:50:05.558848 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:05.558804    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:50:05.558848 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:05.558854    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:50:05.561388 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:05.561361    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f32ae7a-4fbd-4959-a49b-76e9729a5c8e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x9wjc\" (UID: \"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:50:05.561388 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:05.561383    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8aed736-cadf-4091-b78e-1d555860a055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvqh6\" (UID: \"d8aed736-cadf-4091-b78e-1d555860a055\") " pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:50:05.640397 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:05.640369    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"
Oct 30 21:50:05.647886 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:05.647863    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvqh6"
Oct 30 21:50:06.017730 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:06.017705    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc"]
Oct 30 21:50:06.070729 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:06.070638    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvqh6"]
Oct 30 21:50:06.072941 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:50:06.072918    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8aed736_cadf_4091_b78e_1d555860a055.slice/crio-4a57990c242eae8dd97673068041b187e130cb97d5e0460bc439c4bae4de322f WatchSource:0}: Error finding container 4a57990c242eae8dd97673068041b187e130cb97d5e0460bc439c4bae4de322f: Status 404 returned error can't find the container with id 4a57990c242eae8dd97673068041b187e130cb97d5e0460bc439c4bae4de322f
Oct 30 21:50:06.817498 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:06.817466    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvqh6" event={"ID":"d8aed736-cadf-4091-b78e-1d555860a055","Type":"ContainerStarted","Data":"baf53ee8846155fedcafb268ec9d4ff687e8ddc5badfd10f8d68a5994b3691ca"}
Oct 30 21:50:06.817914 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:06.817533    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvqh6" event={"ID":"d8aed736-cadf-4091-b78e-1d555860a055","Type":"ContainerStarted","Data":"4a57990c242eae8dd97673068041b187e130cb97d5e0460bc439c4bae4de322f"}
Oct 30 21:50:06.818653 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:06.818633    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc" event={"ID":"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e","Type":"ContainerStarted","Data":"44c11a793fd62e06681f717e946f5d4f454f98913ba283d76fb583693abb7542"}
Oct 30 21:50:06.843860 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:06.843833    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:50:07.825655 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:07.825612    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvqh6" event={"ID":"d8aed736-cadf-4091-b78e-1d555860a055","Type":"ContainerStarted","Data":"393a08a7e8b628916fefee6e54285dab709e891011bc2f3956d278fc3e757fc4"}
Oct 30 21:50:08.831980 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:08.831942    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc" event={"ID":"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e","Type":"ContainerStarted","Data":"6342184ab1342c11f0b71a6d416eff3ae9b91bbcfd49ab146e78778f0644ee03"}
Oct 30 21:50:08.832473 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:08.832001    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x9wjc" event={"ID":"8f32ae7a-4fbd-4959-a49b-76e9729a5c8e","Type":"ContainerStarted","Data":"aa2a7f95863876ccca5aac0763c5f8d0ad990a81d8ffc6c1678bb931a6740b10"}
Oct 30 21:50:09.838738 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:09.838693    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvqh6" event={"ID":"d8aed736-cadf-4091-b78e-1d555860a055","Type":"ContainerStarted","Data":"a848731712551570cee1543c7f03b9ddbb6ca0a04f40b2e24b540ca97697bfbc"}
Oct 30 21:50:09.982270 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:09.982240    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59f787ddb6-57tbv"]
Oct 30 21:50:10.141548 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:10.141514    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:10.172003 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:10.171975    5031 patch_prober.go:29] interesting pod/image-registry-8698788495-986ld container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:50:10.172143 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:10.172019    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8698788495-986ld" podUID="2e458ae3-5f11-4650-ae5e-0145b063be66" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:50:10.470197 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:10.470125    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:10.736610 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:10.736539    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:10.799309 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:10.799286    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5xrdj"
Oct 30 21:50:11.056493 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:11.056422    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:11.321499 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:11.321438    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:13.102080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:13.102055    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6kj"
Oct 30 21:50:13.779147 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:13.779119    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:14.091868 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:14.091804    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:14.326685 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:14.326649    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:15.713861 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:15.713789    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:16.847742 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:16.847685    5031 patch_prober.go:29] interesting pod/image-registry-8698788495-986ld container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:50:16.848123 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:16.847761    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8698788495-986ld" podUID="2e458ae3-5f11-4650-ae5e-0145b063be66" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:50:18.401496 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:18.401467    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d77b4"
Oct 30 21:50:18.727148 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:18.727070    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:19.441611 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:19.441554    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:19.987394 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:19.987368    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:50:20.048246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.048195    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:20.293272 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.293158    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:20.526832 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.526798    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:20.540651 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:50:20.540623    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:50:20.732401 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.732369    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ssbjl"]
Oct 30 21:50:20.755263 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.755241    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.774281 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.774255    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:20.865807 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.865777    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:50:20.865807 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.865796    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:50:20.865971 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.865824    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:50:20.865971 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.865900    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5k6j\""
Oct 30 21:50:20.865971 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.865924    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:50:20.870514 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870485    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870609 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870527    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79a31e89-7e08-4133-aa3c-4b254b273452-metrics-client-ca\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870609 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870558    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpzd\" (UniqueName: \"kubernetes.io/projected/79a31e89-7e08-4133-aa3c-4b254b273452-kube-api-access-mbpzd\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870680 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870649    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-tls\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870734 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870699    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-sys\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870772 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870743    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-root\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870827 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870809    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870883 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870862    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-wtmp\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.870937 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.870892    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-textfile\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971597 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971570    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-sys\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971597 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971598    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-root\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971633    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971661    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-wtmp\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971678    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-textfile\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971675    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-root\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971675    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-sys\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971723    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971813    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-wtmp\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.971836 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971832    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79a31e89-7e08-4133-aa3c-4b254b273452-metrics-client-ca\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.972240 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971870    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpzd\" (UniqueName: \"kubernetes.io/projected/79a31e89-7e08-4133-aa3c-4b254b273452-kube-api-access-mbpzd\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.972240 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.971919    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-tls\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.972240 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.972016    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-textfile\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.972395 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.972290    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.972503 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.972478    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79a31e89-7e08-4133-aa3c-4b254b273452-metrics-client-ca\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.974124 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.974104    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:20.974452 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:20.974434    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79a31e89-7e08-4133-aa3c-4b254b273452-node-exporter-tls\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:21.035447 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:21.035391    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpzd\" (UniqueName: \"kubernetes.io/projected/79a31e89-7e08-4133-aa3c-4b254b273452-kube-api-access-mbpzd\") pod \"node-exporter-ssbjl\" (UID: \"79a31e89-7e08-4133-aa3c-4b254b273452\") " pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:21.066619 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:21.066592    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:21.104264 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:21.104240    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssbjl"
Oct 30 21:50:21.144440 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:50:21.144414    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a31e89_7e08_4133_aa3c_4b254b273452.slice/crio-da3ab21d50c5150120be721e67669a0833e51844c9c68a31c1e8cf1e0bff3ea4 WatchSource:0}: Error finding container da3ab21d50c5150120be721e67669a0833e51844c9c68a31c1e8cf1e0bff3ea4: Status 404 returned error can't find the container with id da3ab21d50c5150120be721e67669a0833e51844c9c68a31c1e8cf1e0bff3ea4
Oct 30 21:50:21.287316 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:21.287248    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:21.522351 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:21.522320    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:21.885467 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:21.885436    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssbjl" event={"ID":"79a31e89-7e08-4133-aa3c-4b254b273452","Type":"ContainerStarted","Data":"da3ab21d50c5150120be721e67669a0833e51844c9c68a31c1e8cf1e0bff3ea4"}
Oct 30 21:50:22.187139 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:22.187064    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:22.435019 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:22.434990    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:23.132961 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:23.132936    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cjd7c"
Oct 30 21:50:23.150623 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:23.150601    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:23.894383 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:23.894354    5031 generic.go:356] "Generic (PLEG): container finished" podID="79a31e89-7e08-4133-aa3c-4b254b273452" containerID="9560ec6e6d38359579afdc7f0cc38564e4ef5db808fcfaddb07ef25b56217e1c" exitCode=0
Oct 30 21:50:23.894383 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:23.894386    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssbjl" event={"ID":"79a31e89-7e08-4133-aa3c-4b254b273452","Type":"ContainerDied","Data":"9560ec6e6d38359579afdc7f0cc38564e4ef5db808fcfaddb07ef25b56217e1c"}
Oct 30 21:50:24.551620 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:24.551591    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:24.806080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:24.806006    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:24.900281 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:24.900250    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssbjl" event={"ID":"79a31e89-7e08-4133-aa3c-4b254b273452","Type":"ContainerStarted","Data":"0c19114461accbbbdb89552f954dc6ffee244884c33f62046e949999c5723472"}
Oct 30 21:50:24.900416 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:24.900305    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssbjl" event={"ID":"79a31e89-7e08-4133-aa3c-4b254b273452","Type":"ContainerStarted","Data":"9551efd659ad3367cba8945852e378724a0b8a0afd1eb7ef8b78c30d75157198"}
Oct 30 21:50:25.073624 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:25.073543    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ssbjl" podStartSLOduration=3.153835104 podStartE2EDuration="5.073531359s" totalImagesPullingTime="1.834774012s" totalInitContainerRuntime="84.922243ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:25.073028883 +0000 UTC m=+36.597171646" watchObservedRunningTime="2026-10-30 21:50:25.073531359 +0000 UTC m=+36.597674138"
Oct 30 21:50:25.323899 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:25.323828    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:25.569067 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:25.569034    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:25.800830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:25.800803    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:26.848379 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:26.848348    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:50:27.536045 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:27.535971    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:27.764441 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:27.764410    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:27.999446 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:27.999417    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:30.322132 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:30.322096    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:32.363855 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:32.363824    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:32.586359 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:32.586326    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:32.809286 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:32.809191    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:33.293431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:33.293399    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:33.517692 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:33.517657    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:33.746594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:33.746561    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8698788495-986ld"]
Oct 30 21:50:33.932941 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:33.932902    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:34.172310 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:34.172276    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:34.385583 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:34.385550    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:35.003310 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.003278    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv" podUID="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" containerName="registry" containerID="cri-o://9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff" gracePeriod=30
Oct 30 21:50:35.013048 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.013026    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:35.236098 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.236073    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:35.242381 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.242361    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:50:35.380766 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380743    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.380876 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380777    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.380876 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380803    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.380876 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380846    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.380876 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380865    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.381062 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380887    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.381062 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380936    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.381062 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.380964    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") pod \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\" (UID: \"3faa0473-4e54-4ef1-adf1-17f86a91bdd7\") "
Oct 30 21:50:35.381204 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.381171    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:35.381485 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.381453    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:35.383503 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.383460    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:35.383751 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.383715    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:35.383842 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.383747    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:35.383907 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.383842    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:35.383907 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.383874    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "kube-api-access-7l8n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:35.389856 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.389819    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted" pod "3faa0473-4e54-4ef1-adf1-17f86a91bdd7" (UID: "3faa0473-4e54-4ef1-adf1-17f86a91bdd7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:35.481960 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.481931    5031 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-bound-sa-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.481960 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.481953    5031 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-trusted-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.481960 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.481963    5031 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-image-registry-private-configuration\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.482127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.481974    5031 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-ca-trust-extracted\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.482127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.481982    5031 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-certificates\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.482127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.481991    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8n9\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-kube-api-access-7l8n9\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.482127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.481999    5031 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-installation-pull-secrets\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.482127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.482007    5031 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3faa0473-4e54-4ef1-adf1-17f86a91bdd7-registry-tls\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:35.945230 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.945186    5031 generic.go:356] "Generic (PLEG): container finished" podID="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" containerID="9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff" exitCode=0
Oct 30 21:50:35.945431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.945241    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv" event={"ID":"3faa0473-4e54-4ef1-adf1-17f86a91bdd7","Type":"ContainerDied","Data":"9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff"}
Oct 30 21:50:35.945431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.945279    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv"
Oct 30 21:50:35.945431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.945292    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f787ddb6-57tbv" event={"ID":"3faa0473-4e54-4ef1-adf1-17f86a91bdd7","Type":"ContainerDied","Data":"a14a2884ee18d61d6ff5e674afc1b87690be6aad64fd58a96c9fd511dadc61a8"}
Oct 30 21:50:35.945431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.945318    5031 scope.go:122] "RemoveContainer" containerID="9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff"
Oct 30 21:50:35.954672 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.954654    5031 scope.go:122] "RemoveContainer" containerID="9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff"
Oct 30 21:50:35.954994 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:50:35.954970    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff\": container with ID starting with 9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff not found: ID does not exist" containerID="9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff"
Oct 30 21:50:35.955092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:35.955000    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff"} err="failed to get container status \"9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff\": rpc error: code = NotFound desc = could not find container \"9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff\": container with ID starting with 9414b00fd556d4eb0fd890964e9940e25506e5576ae9bd5a3585efdfb7d275ff not found: ID does not exist"
Oct 30 21:50:36.113791 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:36.113764    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59f787ddb6-57tbv"]
Oct 30 21:50:36.164426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:36.164400    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59f787ddb6-57tbv"]
Oct 30 21:50:36.394151 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:36.394119    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:36.613375 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:36.613346    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:36.642197 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:36.642162    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" path="/var/lib/kubelet/pods/3faa0473-4e54-4ef1-adf1-17f86a91bdd7/volumes"
Oct 30 21:50:36.919374 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:36.919344    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:37.141322 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:37.141295    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:38.666970 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:38.666938    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:38.935250 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:38.935162    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:42.863891 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:42.863854    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:43.094588 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:43.094559    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:50.538757 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:50:50.538719    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:50:50.986246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:50.986190    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:51.230563 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:51.230531    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:57.882276 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:57.882237    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:58.099639 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:58.099607    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:58.769674 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:58.769614    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8698788495-986ld" podUID="2e458ae3-5f11-4650-ae5e-0145b063be66" containerName="registry" containerID="cri-o://345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb" gracePeriod=30
Oct 30 21:50:59.027011 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.026955    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:50:59.034573 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.034549    5031 generic.go:356] "Generic (PLEG): container finished" podID="2e458ae3-5f11-4650-ae5e-0145b063be66" containerID="345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb" exitCode=0
Oct 30 21:50:59.034691 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.034588    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698788495-986ld"
Oct 30 21:50:59.034691 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.034618    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698788495-986ld" event={"ID":"2e458ae3-5f11-4650-ae5e-0145b063be66","Type":"ContainerDied","Data":"345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb"}
Oct 30 21:50:59.034691 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.034658    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698788495-986ld" event={"ID":"2e458ae3-5f11-4650-ae5e-0145b063be66","Type":"ContainerDied","Data":"6c6ff2cd9a00606e5667af426db29ced97fde5c724ef86bb954d735cc3c7ec2d"}
Oct 30 21:50:59.034691 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.034676    5031 scope.go:122] "RemoveContainer" containerID="345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb"
Oct 30 21:50:59.042389 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.042366    5031 scope.go:122] "RemoveContainer" containerID="345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb"
Oct 30 21:50:59.042634 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:50:59.042617    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb\": container with ID starting with 345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb not found: ID does not exist" containerID="345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb"
Oct 30 21:50:59.042681 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.042644    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb"} err="failed to get container status \"345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb\": rpc error: code = NotFound desc = could not find container \"345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb\": container with ID starting with 345327c7b0613e572cc312b1d0e56436841266d71d5385cbbaf55f397236e5fb not found: ID does not exist"
Oct 30 21:50:59.050896 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.050880    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.050966 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.050913    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.050966 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.050938    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.051033 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.050975    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.051033 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.051020    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.051127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.051051    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.051127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.051089    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.051233 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.051141    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") pod \"2e458ae3-5f11-4650-ae5e-0145b063be66\" (UID: \"2e458ae3-5f11-4650-ae5e-0145b063be66\") "
Oct 30 21:50:59.052377 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.052109    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:59.054002 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.053963    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:59.054142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.054120    5031 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-tls\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.054233 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.054142    5031 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-trusted-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.054470 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.054439    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:59.054677 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.054654    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:59.054808 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.054747    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "kube-api-access-6mrs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:59.054808 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.054761    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:59.058451 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.057479    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:59.062021 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.061995    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted" pod "2e458ae3-5f11-4650-ae5e-0145b063be66" (UID: "2e458ae3-5f11-4650-ae5e-0145b063be66"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:59.154508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.154478    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mrs4\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-kube-api-access-6mrs4\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.154508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.154502    5031 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-installation-pull-secrets\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.154508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.154512    5031 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e458ae3-5f11-4650-ae5e-0145b063be66-image-registry-private-configuration\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.154732 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.154523    5031 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e458ae3-5f11-4650-ae5e-0145b063be66-bound-sa-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.154732 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.154532    5031 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e458ae3-5f11-4650-ae5e-0145b063be66-ca-trust-extracted\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.154732 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.154541    5031 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e458ae3-5f11-4650-ae5e-0145b063be66-registry-certificates\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.459539 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.459504    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8698788495-986ld"]
Oct 30 21:50:59.465882 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:50:59.465860    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8698788495-986ld"]
Oct 30 21:51:00.644622 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:00.644589    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e458ae3-5f11-4650-ae5e-0145b063be66" path="/var/lib/kubelet/pods/2e458ae3-5f11-4650-ae5e-0145b063be66/volumes"
Oct 30 21:51:05.060085 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:05.060001    5031 generic.go:356] "Generic (PLEG): container finished" podID="4775f051-dc74-47a3-b528-a735daac1287" containerID="4a387337f1110c4df6ceec9c6fb732539ecdd51215bd4cb8428fc37267f92c21" exitCode=0
Oct 30 21:51:05.060085 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:05.060067    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r" event={"ID":"4775f051-dc74-47a3-b528-a735daac1287","Type":"ContainerDied","Data":"4a387337f1110c4df6ceec9c6fb732539ecdd51215bd4cb8428fc37267f92c21"}
Oct 30 21:51:05.060565 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:05.060385    5031 scope.go:122] "RemoveContainer" containerID="4a387337f1110c4df6ceec9c6fb732539ecdd51215bd4cb8428fc37267f92c21"
Oct 30 21:51:06.066190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:06.066152    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bz59r" event={"ID":"4775f051-dc74-47a3-b528-a735daac1287","Type":"ContainerStarted","Data":"0dcecda0543fc5f490da24e9f4b91d33eb13bf54b3c8c9d993d14f6e07848b74"}
Oct 30 21:51:08.672143 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:08.672110    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:08.896420 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:08.896385    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:10.077614 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:10.077579    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:10.301180 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:10.301142    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:10.523659 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:10.523629    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:11.468192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:11.468165    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:11.691191 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:11.691162    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:20.119899 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:20.119864    5031 generic.go:356] "Generic (PLEG): container finished" podID="a7c89d90-bbaa-4f37-acb6-c83e561a1d72" containerID="21aa16c98a1df39f01c26c0231013e5ac3be15691d0d87183c3bcf5b9bc32117" exitCode=0
Oct 30 21:51:20.120264 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:20.119938    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s" event={"ID":"a7c89d90-bbaa-4f37-acb6-c83e561a1d72","Type":"ContainerDied","Data":"21aa16c98a1df39f01c26c0231013e5ac3be15691d0d87183c3bcf5b9bc32117"}
Oct 30 21:51:20.120264 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:20.120228    5031 scope.go:122] "RemoveContainer" containerID="21aa16c98a1df39f01c26c0231013e5ac3be15691d0d87183c3bcf5b9bc32117"
Oct 30 21:51:20.539663 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:51:20.539582    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:51:21.125982 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:21.125954    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9994s" event={"ID":"a7c89d90-bbaa-4f37-acb6-c83e561a1d72","Type":"ContainerStarted","Data":"1049d8d40b9ef56b213b77b9f746c0adfa678b7273860b94d17de14b10b37f61"}
Oct 30 21:51:24.139587 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:24.139555    5031 generic.go:356] "Generic (PLEG): container finished" podID="e84861d0-c9a6-480b-82ed-5a0ce96dab7d" containerID="44eb39b26537d74fd985f27f07600555405efe3898eaf919ab4393c71e62313e" exitCode=0
Oct 30 21:51:24.139952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:24.139626    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xh5b2" event={"ID":"e84861d0-c9a6-480b-82ed-5a0ce96dab7d","Type":"ContainerDied","Data":"44eb39b26537d74fd985f27f07600555405efe3898eaf919ab4393c71e62313e"}
Oct 30 21:51:24.139952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:24.139914    5031 scope.go:122] "RemoveContainer" containerID="44eb39b26537d74fd985f27f07600555405efe3898eaf919ab4393c71e62313e"
Oct 30 21:51:25.145611 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.145577    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xh5b2" event={"ID":"e84861d0-c9a6-480b-82ed-5a0ce96dab7d","Type":"ContainerStarted","Data":"792a0f120738dc68007c87637334f7bff5de471f987bc73a0445efa409e48165"}
Oct 30 21:51:25.157640 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.157614    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-scxcw-zq8dc"]
Oct 30 21:51:25.157784 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.157766    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-scxcw-zq8dc" podUID="62848175-7827-47e2-af47-7edd2af96935" containerName="insights-gathering" containerID="cri-o://172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c" gracePeriod=30
Oct 30 21:51:25.426694 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.426668    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:51:25.548472 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548442    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\") pod \"62848175-7827-47e2-af47-7edd2af96935\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") "
Oct 30 21:51:25.548621 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548485    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\") pod \"62848175-7827-47e2-af47-7edd2af96935\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") "
Oct 30 21:51:25.548621 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548545    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\") pod \"62848175-7827-47e2-af47-7edd2af96935\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") "
Oct 30 21:51:25.548621 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548593    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\" (UniqueName: \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\") pod \"62848175-7827-47e2-af47-7edd2af96935\" (UID: \"62848175-7827-47e2-af47-7edd2af96935\") "
Oct 30 21:51:25.548819 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548797    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path" pod "62848175-7827-47e2-af47-7edd2af96935" (UID: "62848175-7827-47e2-af47-7edd2af96935"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:25.548961 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548925    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle" pod "62848175-7827-47e2-af47-7edd2af96935" (UID: "62848175-7827-47e2-af47-7edd2af96935"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:25.548961 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548939    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle" pod "62848175-7827-47e2-af47-7edd2af96935" (UID: "62848175-7827-47e2-af47-7edd2af96935"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:25.548961 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.548940    5031 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62848175-7827-47e2-af47-7edd2af96935-archives-path\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:25.551279 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.551248    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47" pod "62848175-7827-47e2-af47-7edd2af96935" (UID: "62848175-7827-47e2-af47-7edd2af96935"). InnerVolumeSpecName "kube-api-access-f7q47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:25.649973 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.649938    5031 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-trusted-ca-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:25.649973 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.649969    5031 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62848175-7827-47e2-af47-7edd2af96935-service-ca-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:25.649973 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:25.649980    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7q47\" (UniqueName: \"kubernetes.io/projected/62848175-7827-47e2-af47-7edd2af96935-kube-api-access-f7q47\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:26.150182 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.150145    5031 generic.go:356] "Generic (PLEG): container finished" podID="62848175-7827-47e2-af47-7edd2af96935" containerID="172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c" exitCode=2
Oct 30 21:51:26.150594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.150194    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-scxcw-zq8dc"
Oct 30 21:51:26.150594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.150241    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-scxcw-zq8dc" event={"ID":"62848175-7827-47e2-af47-7edd2af96935","Type":"ContainerDied","Data":"172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c"}
Oct 30 21:51:26.150594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.150299    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-scxcw-zq8dc" event={"ID":"62848175-7827-47e2-af47-7edd2af96935","Type":"ContainerDied","Data":"00c6e38ab33fb7aece5c43a27814c8b1346213d902705aa82c45257cc6a500f3"}
Oct 30 21:51:26.150594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.150319    5031 scope.go:122] "RemoveContainer" containerID="172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c"
Oct 30 21:51:26.158928 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.158742    5031 scope.go:122] "RemoveContainer" containerID="172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c"
Oct 30 21:51:26.159043 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:51:26.159024    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c\": container with ID starting with 172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c not found: ID does not exist" containerID="172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c"
Oct 30 21:51:26.159085 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.159052    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c"} err="failed to get container status \"172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c\": rpc error: code = NotFound desc = could not find container \"172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c\": container with ID starting with 172e64490c6348a4b82a16075dd55c6250359f1c234882c9c6a258518e9d687c not found: ID does not exist"
Oct 30 21:51:26.324360 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.324329    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-scxcw-zq8dc"]
Oct 30 21:51:26.329527 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.329502    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-scxcw-zq8dc"]
Oct 30 21:51:26.642755 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:26.642728    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62848175-7827-47e2-af47-7edd2af96935" path="/var/lib/kubelet/pods/62848175-7827-47e2-af47-7edd2af96935/volumes"
Oct 30 21:51:27.878187 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:27.878127    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:28.091793 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:28.091733    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:37.076796 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:37.076762    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:38.709444 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:38.709414    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:38.945446 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:38.945413    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:39.171222 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:51:39.171165    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:50.539846 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:51:50.539811    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:52:20.540637 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:52:20.540562    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:52:39.372371 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:39.372331    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.540422 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:52:50.540382    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:52:50.749500 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:50.749465    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.993337 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:50.993307    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:52.484887 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:52.484856    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:52.717288 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:52.717258    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:53.689116 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689085    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"]
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689400    5031 cpu_manager.go:414] "containerMap: removing container" podUID="62848175-7827-47e2-af47-7edd2af96935" containerName="insights-gathering"
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689411    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62848175-7827-47e2-af47-7edd2af96935" containerName="insights-gathering"
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689430    5031 cpu_manager.go:414] "containerMap: removing container" podUID="2e458ae3-5f11-4650-ae5e-0145b063be66" containerName="registry"
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689435    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e458ae3-5f11-4650-ae5e-0145b063be66" containerName="registry"
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689455    5031 cpu_manager.go:414] "containerMap: removing container" podUID="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" containerName="registry"
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689460    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" containerName="registry"
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689507    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="62848175-7827-47e2-af47-7edd2af96935" containerName="insights-gathering"
Oct 30 21:52:53.689511 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689516    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e458ae3-5f11-4650-ae5e-0145b063be66" containerName="registry"
Oct 30 21:52:53.689797 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.689525    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="3faa0473-4e54-4ef1-adf1-17f86a91bdd7" containerName="registry"
Oct 30 21:52:53.692776 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.692755    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.747862 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.747828    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:52:53.747862 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.747838    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:52:53.797295 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.797264    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-s45cg\""
Oct 30 21:52:53.798525 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.798500    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"]
Oct 30 21:52:53.799376 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.799355    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"]
Oct 30 21:52:53.802828 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.802809    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"]
Oct 30 21:52:53.802960 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.802944    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:53.806064 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.806045    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:53.811239 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.811197    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.811334 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.811315    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr48x\" (UniqueName: \"kubernetes.io/projected/415cebd3-87d4-49aa-8e02-27fcc02b0b29-kube-api-access-mr48x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.811389 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.811376    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.856672 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.856636    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-n5wgr\""
Oct 30 21:52:53.859557 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.859539    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-t7vbt\""
Oct 30 21:52:53.865352 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.865328    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"]
Oct 30 21:52:53.871162 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.871134    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"]
Oct 30 21:52:53.894333 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.894307    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"]
Oct 30 21:52:53.897752 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.897737    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:53.912077 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912057    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.912202 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912097    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr48x\" (UniqueName: \"kubernetes.io/projected/415cebd3-87d4-49aa-8e02-27fcc02b0b29-kube-api-access-mr48x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.912202 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912124    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:53.912202 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912144    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/700db79a-2646-4811-b810-12df4108dfec-kube-api-access-872rp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:53.912202 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912162    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6hz\" (UniqueName: \"kubernetes.io/projected/361999d0-8e99-4235-be6f-9702118856a4-kube-api-access-4d6hz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:53.912469 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912297    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:53.912469 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912338    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.912469 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912362    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:53.912469 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912451    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:53.912623 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912505    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.912660 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.912621    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.944001 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.943923    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:53.952484 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.952456    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xqptj\""
Oct 30 21:52:53.957409 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.957383    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"]
Oct 30 21:52:53.980981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.980953    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr48x\" (UniqueName: \"kubernetes.io/projected/415cebd3-87d4-49aa-8e02-27fcc02b0b29-kube-api-access-mr48x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:53.989671 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.989644    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"]
Oct 30 21:52:53.993099 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:53.993083    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.008671 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.008646    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:52:54.013517 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013498    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:54.013618 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013525    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:54.013618 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013555    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.013618 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013597    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bpgm\" (UniqueName: \"kubernetes.io/projected/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-kube-api-access-8bpgm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.013791 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013687    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:54.013791 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013711    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/700db79a-2646-4811-b810-12df4108dfec-kube-api-access-872rp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:54.013791 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013730    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6hz\" (UniqueName: \"kubernetes.io/projected/361999d0-8e99-4235-be6f-9702118856a4-kube-api-access-4d6hz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:54.013791 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013784    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:54.013995 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013811    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.013995 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013920    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:54.013995 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.013932    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:54.014165 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.014024    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:54.014243 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.014173    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:54.052640 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.052610    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-qz8st\""
Oct 30 21:52:54.053654 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.053626    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"]
Oct 30 21:52:54.079354 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.079290    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6hz\" (UniqueName: \"kubernetes.io/projected/361999d0-8e99-4235-be6f-9702118856a4-kube-api-access-4d6hz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:54.081114 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.081055    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/700db79a-2646-4811-b810-12df4108dfec-kube-api-access-872rp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:54.114196 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114167    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.114326 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114217    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njld6\" (UniqueName: \"kubernetes.io/projected/a0572803-97c0-4ae7-805d-1ca9503a5385-kube-api-access-njld6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.114326 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114250    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bpgm\" (UniqueName: \"kubernetes.io/projected/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-kube-api-access-8bpgm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.114326 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114272    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.114326 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114307    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.114470 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114365    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.114605 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114585    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.114680 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.114663    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.120141 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.120123    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:52:54.128445 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.128423    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:52:54.182680 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.182617    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bpgm\" (UniqueName: \"kubernetes.io/projected/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-kube-api-access-8bpgm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.212663 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.212637    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:52:54.214971 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.214941    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.215099 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.215017    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njld6\" (UniqueName: \"kubernetes.io/projected/a0572803-97c0-4ae7-805d-1ca9503a5385-kube-api-access-njld6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.215099 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.215073    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.215457 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.215432    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.215649 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.215632    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.279043 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.279013    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njld6\" (UniqueName: \"kubernetes.io/projected/a0572803-97c0-4ae7-805d-1ca9503a5385-kube-api-access-njld6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.309007 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.308966    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:52:54.388948 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.388919    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"]
Oct 30 21:52:54.390484 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:52:54.390454    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415cebd3_87d4_49aa_8e02_27fcc02b0b29.slice/crio-9e60669e945a0b9e90367519dfabc805df6ffba50a8931c7e474841a25bb3cc5 WatchSource:0}: Error finding container 9e60669e945a0b9e90367519dfabc805df6ffba50a8931c7e474841a25bb3cc5: Status 404 returned error can't find the container with id 9e60669e945a0b9e90367519dfabc805df6ffba50a8931c7e474841a25bb3cc5
Oct 30 21:52:54.473866 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.473788    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw" event={"ID":"415cebd3-87d4-49aa-8e02-27fcc02b0b29","Type":"ContainerStarted","Data":"9e60669e945a0b9e90367519dfabc805df6ffba50a8931c7e474841a25bb3cc5"}
Oct 30 21:52:54.498043 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.498018    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"]
Oct 30 21:52:54.500276 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:52:54.500245    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700db79a_2646_4811_b810_12df4108dfec.slice/crio-22d7b564de9d96af5bdd1f5d46b5540a5bbd01bc3d678ebd9dbf254d96c44ef3 WatchSource:0}: Error finding container 22d7b564de9d96af5bdd1f5d46b5540a5bbd01bc3d678ebd9dbf254d96c44ef3: Status 404 returned error can't find the container with id 22d7b564de9d96af5bdd1f5d46b5540a5bbd01bc3d678ebd9dbf254d96c44ef3
Oct 30 21:52:54.502601 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.502575    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"]
Oct 30 21:52:54.506277 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:52:54.506248    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361999d0_8e99_4235_be6f_9702118856a4.slice/crio-fd1d8b9abf42accdfeaf72e42f25c3ccfaba877a9a2f281d6e692158771b9d04 WatchSource:0}: Error finding container fd1d8b9abf42accdfeaf72e42f25c3ccfaba877a9a2f281d6e692158771b9d04: Status 404 returned error can't find the container with id fd1d8b9abf42accdfeaf72e42f25c3ccfaba877a9a2f281d6e692158771b9d04
Oct 30 21:52:54.588888 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.588866    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"]
Oct 30 21:52:54.591308 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:52:54.591272    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279b4f64_0057_41ee_8b9b_5ebbef2b4aa5.slice/crio-d64d9b9dc58dafc9e2c673b411f839b1ae5d28154090aedaa4531b7aa3c71bcf WatchSource:0}: Error finding container d64d9b9dc58dafc9e2c673b411f839b1ae5d28154090aedaa4531b7aa3c71bcf: Status 404 returned error can't find the container with id d64d9b9dc58dafc9e2c673b411f839b1ae5d28154090aedaa4531b7aa3c71bcf
Oct 30 21:52:54.682927 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:54.682863    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"]
Oct 30 21:52:54.685431 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:52:54.685406    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0572803_97c0_4ae7_805d_1ca9503a5385.slice/crio-9129911d78f2d768dc376900dac0bca868569a0ab157922d55e98e324a7b5688 WatchSource:0}: Error finding container 9129911d78f2d768dc376900dac0bca868569a0ab157922d55e98e324a7b5688: Status 404 returned error can't find the container with id 9129911d78f2d768dc376900dac0bca868569a0ab157922d55e98e324a7b5688
Oct 30 21:52:55.481471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.481388    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss" event={"ID":"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5","Type":"ContainerStarted","Data":"d64d9b9dc58dafc9e2c673b411f839b1ae5d28154090aedaa4531b7aa3c71bcf"}
Oct 30 21:52:55.483487 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.483458    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb" event={"ID":"700db79a-2646-4811-b810-12df4108dfec","Type":"ContainerStarted","Data":"22d7b564de9d96af5bdd1f5d46b5540a5bbd01bc3d678ebd9dbf254d96c44ef3"}
Oct 30 21:52:55.484942 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.484918    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx" event={"ID":"361999d0-8e99-4235-be6f-9702118856a4","Type":"ContainerStarted","Data":"fd1d8b9abf42accdfeaf72e42f25c3ccfaba877a9a2f281d6e692158771b9d04"}
Oct 30 21:52:55.486427 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.486402    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p" event={"ID":"a0572803-97c0-4ae7-805d-1ca9503a5385","Type":"ContainerStarted","Data":"9129911d78f2d768dc376900dac0bca868569a0ab157922d55e98e324a7b5688"}
Oct 30 21:52:55.510199 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.510169    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"]
Oct 30 21:52:55.513808 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.513697    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.568516 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.568471    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-s8n7b\""
Oct 30 21:52:55.619856 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.619792    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"]
Oct 30 21:52:55.629438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.626899    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.629438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.627023    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.629438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.627149    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcll\" (UniqueName: \"kubernetes.io/projected/62e9baa1-8db0-420a-9e51-26b5a738eaff-kube-api-access-zkcll\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.728742 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.728697    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.728934 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.728795    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.728934 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.728839    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcll\" (UniqueName: \"kubernetes.io/projected/62e9baa1-8db0-420a-9e51-26b5a738eaff-kube-api-access-zkcll\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.729607 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.729580    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.729846 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.729826    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.802279 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.797345    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcll\" (UniqueName: \"kubernetes.io/projected/62e9baa1-8db0-420a-9e51-26b5a738eaff-kube-api-access-zkcll\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:55.834547 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:55.834291    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:52:56.241314 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:56.241276    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"]
Oct 30 21:52:56.248800 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:52:56.248769    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e9baa1_8db0_420a_9e51_26b5a738eaff.slice/crio-e17fdadc481d5638cb715f364885c14bc5ab8d2c5492385c1fec5fc5c3c471e7 WatchSource:0}: Error finding container e17fdadc481d5638cb715f364885c14bc5ab8d2c5492385c1fec5fc5c3c471e7: Status 404 returned error can't find the container with id e17fdadc481d5638cb715f364885c14bc5ab8d2c5492385c1fec5fc5c3c471e7
Oct 30 21:52:56.496035 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:56.495948    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss" event={"ID":"62e9baa1-8db0-420a-9e51-26b5a738eaff","Type":"ContainerStarted","Data":"e17fdadc481d5638cb715f364885c14bc5ab8d2c5492385c1fec5fc5c3c471e7"}
Oct 30 21:52:58.406905 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:58.406862    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.645390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:58.645358    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:59.837431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.837394    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"]
Oct 30 21:52:59.841374 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.841348    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.896251 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.896181    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gbj8s\""
Oct 30 21:52:59.914012 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.913939    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:52:59.951129 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.951001    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"]
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.971856    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2775d7-88bd-440d-8034-5a7412a22128-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.971918    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.971999    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xkt\" (UniqueName: \"kubernetes.io/projected/de2775d7-88bd-440d-8034-5a7412a22128-kube-api-access-j2xkt\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.972030    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2775d7-88bd-440d-8034-5a7412a22128-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.972058    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.972081    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.972113    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.972150    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:52:59.972390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:52:59.972173    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2775d7-88bd-440d-8034-5a7412a22128-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.011780 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.011516    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:00.072884 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.072850    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xkt\" (UniqueName: \"kubernetes.io/projected/de2775d7-88bd-440d-8034-5a7412a22128-kube-api-access-j2xkt\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.072894    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2775d7-88bd-440d-8034-5a7412a22128-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.072916    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.072933    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.072954    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.072987    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.073002    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2775d7-88bd-440d-8034-5a7412a22128-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.073028    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2775d7-88bd-440d-8034-5a7412a22128-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073370 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.073052    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.073541 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.073501    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.075237 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.073816    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.075497 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.074628    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.075497 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.074859    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.076642 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.076586    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2775d7-88bd-440d-8034-5a7412a22128-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.095805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.095776    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2775d7-88bd-440d-8034-5a7412a22128-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.096509 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.096417    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2775d7-88bd-440d-8034-5a7412a22128-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.139223 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.139165    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xkt\" (UniqueName: \"kubernetes.io/projected/de2775d7-88bd-440d-8034-5a7412a22128-kube-api-access-j2xkt\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.140018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.139997    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2775d7-88bd-440d-8034-5a7412a22128-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-kkskg\" (UID: \"de2775d7-88bd-440d-8034-5a7412a22128\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.160028 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.159993    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:00.171730 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.170409    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"]
Oct 30 21:53:00.174568 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.173952    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.231655 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.231625    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-c8gmd\""
Oct 30 21:53:00.240911 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.240876    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"]
Oct 30 21:53:00.250117 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.250085    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:00.274813 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274780    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869e56ec-0645-4744-9763-6c8548f77ad8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274821    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274845    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ldq\" (UniqueName: \"kubernetes.io/projected/869e56ec-0645-4744-9763-6c8548f77ad8-kube-api-access-t6ldq\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274872    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274892    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869e56ec-0645-4744-9763-6c8548f77ad8-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274925    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869e56ec-0645-4744-9763-6c8548f77ad8-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274941    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274960    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.274981 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.274981    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.376064 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.375973    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869e56ec-0645-4744-9763-6c8548f77ad8-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.376064 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.376033    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.376334 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.376069    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.376334 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.376103    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.376334 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.376174    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869e56ec-0645-4744-9763-6c8548f77ad8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.376787 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.376760    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.377415 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.377388    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.379296 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.379270    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869e56ec-0645-4744-9763-6c8548f77ad8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.379397 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.376219    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.379462 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.379412    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ldq\" (UniqueName: \"kubernetes.io/projected/869e56ec-0645-4744-9763-6c8548f77ad8-kube-api-access-t6ldq\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.379519 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.379460    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.379519 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.379491    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869e56ec-0645-4744-9763-6c8548f77ad8-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.380610 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.380588    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.380874 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.380852    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.381686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.381650    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869e56ec-0645-4744-9763-6c8548f77ad8-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.384581 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.384535    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869e56ec-0645-4744-9763-6c8548f77ad8-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.445325 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.445289    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ldq\" (UniqueName: \"kubernetes.io/projected/869e56ec-0645-4744-9763-6c8548f77ad8-kube-api-access-t6ldq\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.456371 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.456307    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869e56ec-0645-4744-9763-6c8548f77ad8-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d\" (UID: \"869e56ec-0645-4744-9763-6c8548f77ad8\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.477180 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.477143    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:00.493490 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.493159    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:00.679772 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.678954    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"]
Oct 30 21:53:00.681940 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:53:00.681913    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2775d7_88bd_440d_8034_5a7412a22128.slice/crio-2f526822a207f1b13ccfc7458f14ff093c67fb4ace8f054296eae23b5a21ae19 WatchSource:0}: Error finding container 2f526822a207f1b13ccfc7458f14ff093c67fb4ace8f054296eae23b5a21ae19: Status 404 returned error can't find the container with id 2f526822a207f1b13ccfc7458f14ff093c67fb4ace8f054296eae23b5a21ae19
Oct 30 21:53:00.941286 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:00.937947    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"]
Oct 30 21:53:00.944284 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:53:00.944252    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869e56ec_0645_4744_9763_6c8548f77ad8.slice/crio-f3defaeb87eb38811088751578ace333c69e399f8efd39e6f48a2a93a2bda443 WatchSource:0}: Error finding container f3defaeb87eb38811088751578ace333c69e399f8efd39e6f48a2a93a2bda443: Status 404 returned error can't find the container with id f3defaeb87eb38811088751578ace333c69e399f8efd39e6f48a2a93a2bda443
Oct 30 21:53:01.525977 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.525944    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"]
Oct 30 21:53:01.532102 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.532075    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.550066 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.549846    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" event={"ID":"de2775d7-88bd-440d-8034-5a7412a22128","Type":"ContainerStarted","Data":"2f526822a207f1b13ccfc7458f14ff093c67fb4ace8f054296eae23b5a21ae19"}
Oct 30 21:53:01.552798 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.552690    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d" event={"ID":"869e56ec-0645-4744-9763-6c8548f77ad8","Type":"ContainerStarted","Data":"f3defaeb87eb38811088751578ace333c69e399f8efd39e6f48a2a93a2bda443"}
Oct 30 21:53:01.589994 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.589954    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"]
Oct 30 21:53:01.592410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.592116    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghfz\" (UniqueName: \"kubernetes.io/projected/59f70b6d-06f2-4890-995e-e1df707e9389-kube-api-access-fghfz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.592410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.592249    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.592410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.592292    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.693197 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.693154    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghfz\" (UniqueName: \"kubernetes.io/projected/59f70b6d-06f2-4890-995e-e1df707e9389-kube-api-access-fghfz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.693405 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.693270    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.693405 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.693317    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.693814 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.693791    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.694501 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.694402    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.761925 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.761857    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghfz\" (UniqueName: \"kubernetes.io/projected/59f70b6d-06f2-4890-995e-e1df707e9389-kube-api-access-fghfz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:01.859782 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:01.859713    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:02.306013 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:02.305756    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"]
Oct 30 21:53:02.563839 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:02.563713    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885" event={"ID":"59f70b6d-06f2-4890-995e-e1df707e9389","Type":"ContainerStarted","Data":"4257bfad0cdcd678b16138070f1368e556b3c8c67843ea95f8c4bff4d31ae36d"}
Oct 30 21:53:04.042152 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:04.042050    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:10.090753 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.090714    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:53:10.091348 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.090808    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:53:10.091348 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.090855    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:53:10.173323 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.173281    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:53:10.173450 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.173359    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:53:10.173450 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.173394    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:53:10.651786 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.651748    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerID="e5768d187d185c84f16ffb887769499ce49dfdb8aacd3a3aa4bb181c27f31a81" exitCode=0
Oct 30 21:53:10.652081 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.652052    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p" event={"ID":"a0572803-97c0-4ae7-805d-1ca9503a5385","Type":"ContainerDied","Data":"e5768d187d185c84f16ffb887769499ce49dfdb8aacd3a3aa4bb181c27f31a81"}
Oct 30 21:53:10.654275 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.654245    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d" event={"ID":"869e56ec-0645-4744-9763-6c8548f77ad8","Type":"ContainerStarted","Data":"ead90805ab17224ff754ee82a007abc44e82b540d8edfce0136f6f2a878ba04a"}
Oct 30 21:53:10.656471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.656437    5031 generic.go:356] "Generic (PLEG): container finished" podID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerID="2d8ce371a7f2f8b0b43c3068362eb7773b90db4fb3a9a11cb775d60176aa1978" exitCode=0
Oct 30 21:53:10.656561 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.656510    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw" event={"ID":"415cebd3-87d4-49aa-8e02-27fcc02b0b29","Type":"ContainerDied","Data":"2d8ce371a7f2f8b0b43c3068362eb7773b90db4fb3a9a11cb775d60176aa1978"}
Oct 30 21:53:10.662277 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.661604    5031 generic.go:356] "Generic (PLEG): container finished" podID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerID="dc7fff0ad1df1cc2664fa4eeae30418fd6daed916dc2f6374756a898f2aee277" exitCode=0
Oct 30 21:53:10.662277 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.661892    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss" event={"ID":"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5","Type":"ContainerDied","Data":"dc7fff0ad1df1cc2664fa4eeae30418fd6daed916dc2f6374756a898f2aee277"}
Oct 30 21:53:10.663982 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.663903    5031 generic.go:356] "Generic (PLEG): container finished" podID="700db79a-2646-4811-b810-12df4108dfec" containerID="4a7889885d23803bba6efecce19eae0413242ac5d66d674dc83b8da7b2ac1094" exitCode=0
Oct 30 21:53:10.664089 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.664053    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb" event={"ID":"700db79a-2646-4811-b810-12df4108dfec","Type":"ContainerDied","Data":"4a7889885d23803bba6efecce19eae0413242ac5d66d674dc83b8da7b2ac1094"}
Oct 30 21:53:10.667395 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.666683    5031 generic.go:356] "Generic (PLEG): container finished" podID="361999d0-8e99-4235-be6f-9702118856a4" containerID="eecff24d3d2daff5ec1ecd58152ce10128101eddf9cad0cdb4c4521ad725f2db" exitCode=0
Oct 30 21:53:10.667395 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.666774    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx" event={"ID":"361999d0-8e99-4235-be6f-9702118856a4","Type":"ContainerDied","Data":"eecff24d3d2daff5ec1ecd58152ce10128101eddf9cad0cdb4c4521ad725f2db"}
Oct 30 21:53:10.669564 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:10.669537    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" event={"ID":"de2775d7-88bd-440d-8034-5a7412a22128","Type":"ContainerStarted","Data":"b91146d0fda3ea260fbc46d0c6b6c464e1f3d648e767627ed2a0531c1e35fc0b"}
Oct 30 21:53:11.156096 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.156036    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d" podStartSLOduration=1.931906701 podStartE2EDuration="11.156014151s" totalImagesPullingTime="9.22410745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:11.154413894 +0000 UTC m=+202.678556661" watchObservedRunningTime="2026-10-30 21:53:11.156014151 +0000 UTC m=+202.680156912"
Oct 30 21:53:11.160700 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.160604    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:11.166135 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.166105    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:11.494154 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.494113    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:11.498768 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.498744    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:11.528253 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.528181    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podStartSLOduration=3.125078415 podStartE2EDuration="12.528165728s" totalImagesPullingTime="9.403087313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:11.527926137 +0000 UTC m=+203.052068902" watchObservedRunningTime="2026-10-30 21:53:11.528165728 +0000 UTC m=+203.052308491"
Oct 30 21:53:11.675544 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.675511    5031 generic.go:356] "Generic (PLEG): container finished" podID="59f70b6d-06f2-4890-995e-e1df707e9389" containerID="a71c4a23e3dfec257929370591da054aa9aca0fc1e1d4bb2c1065c5d68880df6" exitCode=0
Oct 30 21:53:11.675736 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.675598    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885" event={"ID":"59f70b6d-06f2-4890-995e-e1df707e9389","Type":"ContainerDied","Data":"a71c4a23e3dfec257929370591da054aa9aca0fc1e1d4bb2c1065c5d68880df6"}
Oct 30 21:53:11.677282 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.677259    5031 generic.go:356] "Generic (PLEG): container finished" podID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerID="710ca59c22fe8f47afad77ee49f6ab9064869d6d42162ed312536288c6231377" exitCode=0
Oct 30 21:53:11.677399 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.677328    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss" event={"ID":"62e9baa1-8db0-420a-9e51-26b5a738eaff","Type":"ContainerDied","Data":"710ca59c22fe8f47afad77ee49f6ab9064869d6d42162ed312536288c6231377"}
Oct 30 21:53:11.677584 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.677571    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:11.677860 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.677785    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:11.679042 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.679024    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d"
Oct 30 21:53:11.679127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:11.679072    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:53:12.684660 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.684610    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerID="42c7b0e81a4513d9cd3a24fce59fd3da3c73232a4b4976e0b61d517d0f4a6ef8" exitCode=0
Oct 30 21:53:12.685131 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.684691    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p" event={"ID":"a0572803-97c0-4ae7-805d-1ca9503a5385","Type":"ContainerDied","Data":"42c7b0e81a4513d9cd3a24fce59fd3da3c73232a4b4976e0b61d517d0f4a6ef8"}
Oct 30 21:53:12.687331 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.687196    5031 generic.go:356] "Generic (PLEG): container finished" podID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerID="21a95661139df62c0370bdc9ffdb4cde52929385a54d91df34134323e081c818" exitCode=0
Oct 30 21:53:12.687331 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.687295    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw" event={"ID":"415cebd3-87d4-49aa-8e02-27fcc02b0b29","Type":"ContainerDied","Data":"21a95661139df62c0370bdc9ffdb4cde52929385a54d91df34134323e081c818"}
Oct 30 21:53:12.689927 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.689841    5031 generic.go:356] "Generic (PLEG): container finished" podID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerID="2db02f2ff1f8f6d2284725dcbbebd5235470c1ee168403e86c1e3b7b630b72d2" exitCode=0
Oct 30 21:53:12.689990 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.689922    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss" event={"ID":"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5","Type":"ContainerDied","Data":"2db02f2ff1f8f6d2284725dcbbebd5235470c1ee168403e86c1e3b7b630b72d2"}
Oct 30 21:53:12.692553 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.692452    5031 generic.go:356] "Generic (PLEG): container finished" podID="361999d0-8e99-4235-be6f-9702118856a4" containerID="a0972fb2b029ae49e38c4ba318d83c60f8f8e91607aae92a2878a44d2a39450b" exitCode=0
Oct 30 21:53:12.692553 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:12.692514    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx" event={"ID":"361999d0-8e99-4235-be6f-9702118856a4","Type":"ContainerDied","Data":"a0972fb2b029ae49e38c4ba318d83c60f8f8e91607aae92a2878a44d2a39450b"}
Oct 30 21:53:14.713577 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:14.713535    5031 generic.go:356] "Generic (PLEG): container finished" podID="700db79a-2646-4811-b810-12df4108dfec" containerID="0ffa5515bbe3c803f22c6c0f180fcaecfed90de3ee58b8cfd3a8fbf7951e4198" exitCode=0
Oct 30 21:53:14.714061 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:14.713642    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb" event={"ID":"700db79a-2646-4811-b810-12df4108dfec","Type":"ContainerDied","Data":"0ffa5515bbe3c803f22c6c0f180fcaecfed90de3ee58b8cfd3a8fbf7951e4198"}
Oct 30 21:53:17.741526 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:17.741486    5031 generic.go:356] "Generic (PLEG): container finished" podID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerID="24552791c7bf20d8c0bf6047bee5d03c2c9f66b74ed0d85ac206540a53c7ba52" exitCode=0
Oct 30 21:53:17.741974 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:17.741573    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss" event={"ID":"62e9baa1-8db0-420a-9e51-26b5a738eaff","Type":"ContainerDied","Data":"24552791c7bf20d8c0bf6047bee5d03c2c9f66b74ed0d85ac206540a53c7ba52"}
Oct 30 21:53:19.760566 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:19.760459    5031 generic.go:356] "Generic (PLEG): container finished" podID="59f70b6d-06f2-4890-995e-e1df707e9389" containerID="36053ab09caef4361c49dbeecc9a6224e7cbc9c91110485f66d582dab323e0c4" exitCode=0
Oct 30 21:53:19.760566 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:19.760521    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885" event={"ID":"59f70b6d-06f2-4890-995e-e1df707e9389","Type":"ContainerDied","Data":"36053ab09caef4361c49dbeecc9a6224e7cbc9c91110485f66d582dab323e0c4"}
Oct 30 21:53:20.559250 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:53:20.558147    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:53:25.684321 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:25.684224    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:26.054113 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.053930    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:26.331969 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.331757    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:26.604804 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.604774    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:26.814546 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.814513    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"]
Oct 30 21:53:26.834885 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.834839    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw" event={"ID":"415cebd3-87d4-49aa-8e02-27fcc02b0b29","Type":"ContainerStarted","Data":"a46708253735d40ca9ec05e453a80385b3886d58e14ae59e583daf69a4ffb0be"}
Oct 30 21:53:26.835054 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.834964    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:26.882664 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.882623    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:26.938088 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.938024    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:53:26.939242 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.938898    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:53:26.939242 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.938976    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:53:26.939242 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.939066    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:53:26.939242 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.939153    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4xvvf\""
Oct 30 21:53:26.939242 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.939199    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:53:26.963318 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.962698    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fad-01c6-4844-8a60-1976734dd5e9-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:26.963318 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.962750    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:26.963318 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.962792    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fad-01c6-4844-8a60-1976734dd5e9-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:26.963318 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.962842    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:26.963318 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.962871    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq529\" (UniqueName: \"kubernetes.io/projected/c2d91fad-01c6-4844-8a60-1976734dd5e9-kube-api-access-fq529\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:26.963318 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.962890    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:26.994851 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.993093    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"]
Oct 30 21:53:26.994851 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:26.993437    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw" podStartSLOduration=2.747086865 podStartE2EDuration="33.993419922s" totalImagesPullingTime="31.246333057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:26.989136968 +0000 UTC m=+218.513279744" watchObservedRunningTime="2026-10-30 21:53:26.993419922 +0000 UTC m=+218.517562683"
Oct 30 21:53:27.064004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.063576    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fad-01c6-4844-8a60-1976734dd5e9-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.064004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.063638    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.064004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.063679    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fad-01c6-4844-8a60-1976734dd5e9-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.064004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.063730    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.064004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.063763    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq529\" (UniqueName: \"kubernetes.io/projected/c2d91fad-01c6-4844-8a60-1976734dd5e9-kube-api-access-fq529\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.064004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.063791    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.067046 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.067019    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fad-01c6-4844-8a60-1976734dd5e9-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.067796 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.067774    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fad-01c6-4844-8a60-1976734dd5e9-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.067910 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.067846    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.070585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.070566    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.073656 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.073580    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2d91fad-01c6-4844-8a60-1976734dd5e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.141156 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.141074    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:27.147446 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.147418    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq529\" (UniqueName: \"kubernetes.io/projected/c2d91fad-01c6-4844-8a60-1976734dd5e9-kube-api-access-fq529\") pod \"opendatahub-operator-controller-manager-77b67b5d96-gchps\" (UID: \"c2d91fad-01c6-4844-8a60-1976734dd5e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.166766 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.166735    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:27.370826 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.370741    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:27.676783 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.676759    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"]
Oct 30 21:53:27.683106 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:53:27.683055    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d91fad_01c6_4844_8a60_1976734dd5e9.slice/crio-cad0cc6485048d7ae36ec11a5ececd5a389a1cd06ee395c55f5babcee671d8d2 WatchSource:0}: Error finding container cad0cc6485048d7ae36ec11a5ececd5a389a1cd06ee395c55f5babcee671d8d2: Status 404 returned error can't find the container with id cad0cc6485048d7ae36ec11a5ececd5a389a1cd06ee395c55f5babcee671d8d2
Oct 30 21:53:27.827421 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.827383    5031 generic.go:356] "Generic (PLEG): container finished" podID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerID="a46708253735d40ca9ec05e453a80385b3886d58e14ae59e583daf69a4ffb0be" exitCode=0
Oct 30 21:53:27.827893 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.827481    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw" event={"ID":"415cebd3-87d4-49aa-8e02-27fcc02b0b29","Type":"ContainerDied","Data":"a46708253735d40ca9ec05e453a80385b3886d58e14ae59e583daf69a4ffb0be"}
Oct 30 21:53:27.830143 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.830117    5031 generic.go:356] "Generic (PLEG): container finished" podID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerID="60271b830dda221d5e75c004debb742514e20c7fce72814e574ade1a8a1aa6ac" exitCode=0
Oct 30 21:53:27.830297 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.830190    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss" event={"ID":"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5","Type":"ContainerDied","Data":"60271b830dda221d5e75c004debb742514e20c7fce72814e574ade1a8a1aa6ac"}
Oct 30 21:53:27.833363 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.833193    5031 generic.go:356] "Generic (PLEG): container finished" podID="361999d0-8e99-4235-be6f-9702118856a4" containerID="a9de1f31976034af274dfabb33942642a97e654c5e314b87127e6ca883b557c1" exitCode=0
Oct 30 21:53:27.833363 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.833351    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx" event={"ID":"361999d0-8e99-4235-be6f-9702118856a4","Type":"ContainerDied","Data":"a9de1f31976034af274dfabb33942642a97e654c5e314b87127e6ca883b557c1"}
Oct 30 21:53:27.836191 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.836165    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerID="9ee93ddcd5d5ec5dd7099f6e5d1d79ef7a18acdefe7364ca5af7e84947fd9b74" exitCode=0
Oct 30 21:53:27.836361 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.836331    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p" event={"ID":"a0572803-97c0-4ae7-805d-1ca9503a5385","Type":"ContainerDied","Data":"9ee93ddcd5d5ec5dd7099f6e5d1d79ef7a18acdefe7364ca5af7e84947fd9b74"}
Oct 30 21:53:27.839043 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.839020    5031 generic.go:356] "Generic (PLEG): container finished" podID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerID="46bc68174be17196183759e0638f0941175ef9fda1c14f541d0eac3eeb9f3aba" exitCode=0
Oct 30 21:53:27.839181 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.839165    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss" event={"ID":"62e9baa1-8db0-420a-9e51-26b5a738eaff","Type":"ContainerDied","Data":"46bc68174be17196183759e0638f0941175ef9fda1c14f541d0eac3eeb9f3aba"}
Oct 30 21:53:27.840764 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:27.840736    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps" event={"ID":"c2d91fad-01c6-4844-8a60-1976734dd5e9","Type":"ContainerStarted","Data":"cad0cc6485048d7ae36ec11a5ececd5a389a1cd06ee395c55f5babcee671d8d2"}
Oct 30 21:53:28.849148 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:28.849116    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb" event={"ID":"700db79a-2646-4811-b810-12df4108dfec","Type":"ContainerStarted","Data":"cd0a24cbe2541833b92b3a17e3cdb0a2dd474916ea816ed0d7fb4ada9dae6887"}
Oct 30 21:53:28.852079 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:28.852043    5031 generic.go:356] "Generic (PLEG): container finished" podID="59f70b6d-06f2-4890-995e-e1df707e9389" containerID="aa0d592bdebee92362d90259c3524c33b19c73af47a9a24372f489a7f2396859" exitCode=0
Oct 30 21:53:28.852545 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:28.852520    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885" event={"ID":"59f70b6d-06f2-4890-995e-e1df707e9389","Type":"ContainerDied","Data":"aa0d592bdebee92362d90259c3524c33b19c73af47a9a24372f489a7f2396859"}
Oct 30 21:53:29.028513 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.028253    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:29.032629 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.032601    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:53:29.053513 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.053071    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb" podStartSLOduration=3.02308679 podStartE2EDuration="36.0530546s" totalImagesPullingTime="33.02996781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:29.05234276 +0000 UTC m=+220.576485612" watchObservedRunningTime="2026-10-30 21:53:29.0530546 +0000 UTC m=+220.577197366"
Oct 30 21:53:29.103681 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.103649    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-util\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-util\") pod \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") "
Oct 30 21:53:29.103805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.103722    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415cebd3-87d4-49aa-8e02-27fcc02b0b29-kube-api-access-mr48x\" (UniqueName: \"kubernetes.io/projected/415cebd3-87d4-49aa-8e02-27fcc02b0b29-kube-api-access-mr48x\") pod \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") "
Oct 30 21:53:29.103805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.103776    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-bundle\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-bundle\") pod \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\" (UID: \"415cebd3-87d4-49aa-8e02-27fcc02b0b29\") "
Oct 30 21:53:29.104663 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.104473    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-bundle" pod "415cebd3-87d4-49aa-8e02-27fcc02b0b29" (UID: "415cebd3-87d4-49aa-8e02-27fcc02b0b29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.107857 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.107834    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415cebd3-87d4-49aa-8e02-27fcc02b0b29-kube-api-access-mr48x" pod "415cebd3-87d4-49aa-8e02-27fcc02b0b29" (UID: "415cebd3-87d4-49aa-8e02-27fcc02b0b29"). InnerVolumeSpecName "kube-api-access-mr48x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:29.114450 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.114419    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-util" pod "415cebd3-87d4-49aa-8e02-27fcc02b0b29" (UID: "415cebd3-87d4-49aa-8e02-27fcc02b0b29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.123435 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.123406    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:53:29.128893 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.128817    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:53:29.130936 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.130906    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:53:29.136919 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.136894    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:53:29.205070 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205042    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0572803-97c0-4ae7-805d-1ca9503a5385-kube-api-access-njld6\" (UniqueName: \"kubernetes.io/projected/a0572803-97c0-4ae7-805d-1ca9503a5385-kube-api-access-njld6\") pod \"a0572803-97c0-4ae7-805d-1ca9503a5385\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") "
Oct 30 21:53:29.205254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205092    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-bundle\") pod \"a0572803-97c0-4ae7-805d-1ca9503a5385\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") "
Oct 30 21:53:29.205254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205125    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-util\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-util\") pod \"361999d0-8e99-4235-be6f-9702118856a4\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") "
Oct 30 21:53:29.205254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205161    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-util\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-util\") pod \"a0572803-97c0-4ae7-805d-1ca9503a5385\" (UID: \"a0572803-97c0-4ae7-805d-1ca9503a5385\") "
Oct 30 21:53:29.205254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205203    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-bundle\") pod \"361999d0-8e99-4235-be6f-9702118856a4\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") "
Oct 30 21:53:29.205254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205253    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-kube-api-access-8bpgm\" (UniqueName: \"kubernetes.io/projected/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-kube-api-access-8bpgm\") pod \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") "
Oct 30 21:53:29.205539 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205290    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361999d0-8e99-4235-be6f-9702118856a4-kube-api-access-4d6hz\" (UniqueName: \"kubernetes.io/projected/361999d0-8e99-4235-be6f-9702118856a4-kube-api-access-4d6hz\") pod \"361999d0-8e99-4235-be6f-9702118856a4\" (UID: \"361999d0-8e99-4235-be6f-9702118856a4\") "
Oct 30 21:53:29.205539 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205327    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e9baa1-8db0-420a-9e51-26b5a738eaff-kube-api-access-zkcll\" (UniqueName: \"kubernetes.io/projected/62e9baa1-8db0-420a-9e51-26b5a738eaff-kube-api-access-zkcll\") pod \"62e9baa1-8db0-420a-9e51-26b5a738eaff\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") "
Oct 30 21:53:29.205539 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205376    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-util\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-util\") pod \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") "
Oct 30 21:53:29.205539 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205409    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-bundle\") pod \"62e9baa1-8db0-420a-9e51-26b5a738eaff\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") "
Oct 30 21:53:29.205746 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205575    5031 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-util\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.205746 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205618    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr48x\" (UniqueName: \"kubernetes.io/projected/415cebd3-87d4-49aa-8e02-27fcc02b0b29-kube-api-access-mr48x\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.205746 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205629    5031 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415cebd3-87d4-49aa-8e02-27fcc02b0b29-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.205976 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.205958    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-bundle" pod "62e9baa1-8db0-420a-9e51-26b5a738eaff" (UID: "62e9baa1-8db0-420a-9e51-26b5a738eaff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.206064 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.206038    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-bundle" pod "a0572803-97c0-4ae7-805d-1ca9503a5385" (UID: "a0572803-97c0-4ae7-805d-1ca9503a5385"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.206595 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.206552    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-bundle" pod "361999d0-8e99-4235-be6f-9702118856a4" (UID: "361999d0-8e99-4235-be6f-9702118856a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.207665 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.207634    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0572803-97c0-4ae7-805d-1ca9503a5385-kube-api-access-njld6" pod "a0572803-97c0-4ae7-805d-1ca9503a5385" (UID: "a0572803-97c0-4ae7-805d-1ca9503a5385"). InnerVolumeSpecName "kube-api-access-njld6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:29.208580 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.208552    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e9baa1-8db0-420a-9e51-26b5a738eaff-kube-api-access-zkcll" pod "62e9baa1-8db0-420a-9e51-26b5a738eaff" (UID: "62e9baa1-8db0-420a-9e51-26b5a738eaff"). InnerVolumeSpecName "kube-api-access-zkcll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:29.208914 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.208886    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-kube-api-access-8bpgm" pod "279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" (UID: "279b4f64-0057-41ee-8b9b-5ebbef2b4aa5"). InnerVolumeSpecName "kube-api-access-8bpgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:29.209245 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.209203    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361999d0-8e99-4235-be6f-9702118856a4-kube-api-access-4d6hz" pod "361999d0-8e99-4235-be6f-9702118856a4" (UID: "361999d0-8e99-4235-be6f-9702118856a4"). InnerVolumeSpecName "kube-api-access-4d6hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:29.210872 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.210848    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-util" pod "361999d0-8e99-4235-be6f-9702118856a4" (UID: "361999d0-8e99-4235-be6f-9702118856a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.212357 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.212330    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-util" pod "a0572803-97c0-4ae7-805d-1ca9503a5385" (UID: "a0572803-97c0-4ae7-805d-1ca9503a5385"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.213596 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.213578    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-util" pod "279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" (UID: "279b4f64-0057-41ee-8b9b-5ebbef2b4aa5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.258486 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.258448    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:29.306168 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306090    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-bundle\") pod \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\" (UID: \"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5\") "
Oct 30 21:53:29.306168 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306133    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-util\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-util\") pod \"62e9baa1-8db0-420a-9e51-26b5a738eaff\" (UID: \"62e9baa1-8db0-420a-9e51-26b5a738eaff\") "
Oct 30 21:53:29.306432 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306399    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6hz\" (UniqueName: \"kubernetes.io/projected/361999d0-8e99-4235-be6f-9702118856a4-kube-api-access-4d6hz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306432 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306420    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcll\" (UniqueName: \"kubernetes.io/projected/62e9baa1-8db0-420a-9e51-26b5a738eaff-kube-api-access-zkcll\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306436    5031 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-util\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306451    5031 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306465    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njld6\" (UniqueName: \"kubernetes.io/projected/a0572803-97c0-4ae7-805d-1ca9503a5385-kube-api-access-njld6\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306477    5031 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306490    5031 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-util\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306502    5031 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0572803-97c0-4ae7-805d-1ca9503a5385-util\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306513    5031 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/361999d0-8e99-4235-be6f-9702118856a4-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306527    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bpgm\" (UniqueName: \"kubernetes.io/projected/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-kube-api-access-8bpgm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.306851 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.306799    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-bundle" pod "279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" (UID: "279b4f64-0057-41ee-8b9b-5ebbef2b4aa5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.313587 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.313554    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-util" pod "62e9baa1-8db0-420a-9e51-26b5a738eaff" (UID: "62e9baa1-8db0-420a-9e51-26b5a738eaff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:29.407867 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.407830    5031 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/279b4f64-0057-41ee-8b9b-5ebbef2b4aa5-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.407867 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.407867    5031 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e9baa1-8db0-420a-9e51-26b5a738eaff-util\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:29.859493 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.859455    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss"
Oct 30 21:53:29.859940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.859456    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448gnss" event={"ID":"62e9baa1-8db0-420a-9e51-26b5a738eaff","Type":"ContainerDied","Data":"e17fdadc481d5638cb715f364885c14bc5ab8d2c5492385c1fec5fc5c3c471e7"}
Oct 30 21:53:29.859940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.859604    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e17fdadc481d5638cb715f364885c14bc5ab8d2c5492385c1fec5fc5c3c471e7"
Oct 30 21:53:29.863649 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.863609    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw" event={"ID":"415cebd3-87d4-49aa-8e02-27fcc02b0b29","Type":"ContainerDied","Data":"9e60669e945a0b9e90367519dfabc805df6ffba50a8931c7e474841a25bb3cc5"}
Oct 30 21:53:29.863804 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.863664    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e60669e945a0b9e90367519dfabc805df6ffba50a8931c7e474841a25bb3cc5"
Oct 30 21:53:29.863804 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.863621    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64bnpw"
Oct 30 21:53:29.866100 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.866080    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss"
Oct 30 21:53:29.866240 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.866077    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c867ss" event={"ID":"279b4f64-0057-41ee-8b9b-5ebbef2b4aa5","Type":"ContainerDied","Data":"d64d9b9dc58dafc9e2c673b411f839b1ae5d28154090aedaa4531b7aa3c71bcf"}
Oct 30 21:53:29.866240 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.866203    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64d9b9dc58dafc9e2c673b411f839b1ae5d28154090aedaa4531b7aa3c71bcf"
Oct 30 21:53:29.869703 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.869616    5031 generic.go:356] "Generic (PLEG): container finished" podID="700db79a-2646-4811-b810-12df4108dfec" containerID="cd0a24cbe2541833b92b3a17e3cdb0a2dd474916ea816ed0d7fb4ada9dae6887" exitCode=0
Oct 30 21:53:29.869855 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.869781    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb" event={"ID":"700db79a-2646-4811-b810-12df4108dfec","Type":"ContainerDied","Data":"cd0a24cbe2541833b92b3a17e3cdb0a2dd474916ea816ed0d7fb4ada9dae6887"}
Oct 30 21:53:29.874106 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.874085    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx"
Oct 30 21:53:29.874232 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.874132    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4kxxx" event={"ID":"361999d0-8e99-4235-be6f-9702118856a4","Type":"ContainerDied","Data":"fd1d8b9abf42accdfeaf72e42f25c3ccfaba877a9a2f281d6e692158771b9d04"}
Oct 30 21:53:29.874232 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.874174    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd1d8b9abf42accdfeaf72e42f25c3ccfaba877a9a2f281d6e692158771b9d04"
Oct 30 21:53:29.876609 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.876586    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p"
Oct 30 21:53:29.876730 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.876584    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vdl2p" event={"ID":"a0572803-97c0-4ae7-805d-1ca9503a5385","Type":"ContainerDied","Data":"9129911d78f2d768dc376900dac0bca868569a0ab157922d55e98e324a7b5688"}
Oct 30 21:53:29.876795 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:29.876749    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9129911d78f2d768dc376900dac0bca868569a0ab157922d55e98e324a7b5688"
Oct 30 21:53:30.023869 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.023845    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:30.114938 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.114905    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-util\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-util\") pod \"59f70b6d-06f2-4890-995e-e1df707e9389\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") "
Oct 30 21:53:30.115135 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.115008    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-bundle\") pod \"59f70b6d-06f2-4890-995e-e1df707e9389\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") "
Oct 30 21:53:30.115135 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.115036    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f70b6d-06f2-4890-995e-e1df707e9389-kube-api-access-fghfz\" (UniqueName: \"kubernetes.io/projected/59f70b6d-06f2-4890-995e-e1df707e9389-kube-api-access-fghfz\") pod \"59f70b6d-06f2-4890-995e-e1df707e9389\" (UID: \"59f70b6d-06f2-4890-995e-e1df707e9389\") "
Oct 30 21:53:30.115546 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.115519    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-bundle" pod "59f70b6d-06f2-4890-995e-e1df707e9389" (UID: "59f70b6d-06f2-4890-995e-e1df707e9389"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:30.117596 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.117570    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f70b6d-06f2-4890-995e-e1df707e9389-kube-api-access-fghfz" pod "59f70b6d-06f2-4890-995e-e1df707e9389" (UID: "59f70b6d-06f2-4890-995e-e1df707e9389"). InnerVolumeSpecName "kube-api-access-fghfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:30.119402 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.119378    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-util" pod "59f70b6d-06f2-4890-995e-e1df707e9389" (UID: "59f70b6d-06f2-4890-995e-e1df707e9389"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:30.216085 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.216053    5031 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-util\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:30.216085 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.216086    5031 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f70b6d-06f2-4890-995e-e1df707e9389-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:30.216338 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.216102    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghfz\" (UniqueName: \"kubernetes.io/projected/59f70b6d-06f2-4890-995e-e1df707e9389-kube-api-access-fghfz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:30.884581 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.884548    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885"
Oct 30 21:53:30.885024 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.884539    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5885" event={"ID":"59f70b6d-06f2-4890-995e-e1df707e9389","Type":"ContainerDied","Data":"4257bfad0cdcd678b16138070f1368e556b3c8c67843ea95f8c4bff4d31ae36d"}
Oct 30 21:53:30.885024 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:30.884686    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4257bfad0cdcd678b16138070f1368e556b3c8c67843ea95f8c4bff4d31ae36d"
Oct 30 21:53:31.026553 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.026523    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:53:31.124939 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.124904    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-bundle\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-bundle\") pod \"700db79a-2646-4811-b810-12df4108dfec\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") "
Oct 30 21:53:31.125128 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.124958    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700db79a-2646-4811-b810-12df4108dfec-kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/700db79a-2646-4811-b810-12df4108dfec-kube-api-access-872rp\") pod \"700db79a-2646-4811-b810-12df4108dfec\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") "
Oct 30 21:53:31.125128 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.125101    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-util\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-util\") pod \"700db79a-2646-4811-b810-12df4108dfec\" (UID: \"700db79a-2646-4811-b810-12df4108dfec\") "
Oct 30 21:53:31.125961 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.125927    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-bundle" pod "700db79a-2646-4811-b810-12df4108dfec" (UID: "700db79a-2646-4811-b810-12df4108dfec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:31.127304 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.127272    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700db79a-2646-4811-b810-12df4108dfec-kube-api-access-872rp" pod "700db79a-2646-4811-b810-12df4108dfec" (UID: "700db79a-2646-4811-b810-12df4108dfec"). InnerVolumeSpecName "kube-api-access-872rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:31.130737 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.130712    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-util" pod "700db79a-2646-4811-b810-12df4108dfec" (UID: "700db79a-2646-4811-b810-12df4108dfec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:31.225902 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.225831    5031 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-util\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:31.225902 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.225859    5031 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700db79a-2646-4811-b810-12df4108dfec-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:31.225902 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.225871    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/700db79a-2646-4811-b810-12df4108dfec-kube-api-access-872rp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:31.890850 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.890818    5031 generic.go:356] "Generic (PLEG): container finished" podID="c2d91fad-01c6-4844-8a60-1976734dd5e9" containerID="38befd18af9c807425005e04c8b97b8b647d8ed311a1282ad009cb925cff80b2" exitCode=0
Oct 30 21:53:31.891269 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.890908    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps" event={"ID":"c2d91fad-01c6-4844-8a60-1976734dd5e9","Type":"ContainerDied","Data":"38befd18af9c807425005e04c8b97b8b647d8ed311a1282ad009cb925cff80b2"}
Oct 30 21:53:31.893145 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.893126    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb"
Oct 30 21:53:31.893276 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.893159    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdm8rb" event={"ID":"700db79a-2646-4811-b810-12df4108dfec","Type":"ContainerDied","Data":"22d7b564de9d96af5bdd1f5d46b5540a5bbd01bc3d678ebd9dbf254d96c44ef3"}
Oct 30 21:53:31.893276 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:31.893230    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22d7b564de9d96af5bdd1f5d46b5540a5bbd01bc3d678ebd9dbf254d96c44ef3"
Oct 30 21:53:33.038447 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:33.038421    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.277169 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:33.277083    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:33.905150 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:33.905118    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps" event={"ID":"c2d91fad-01c6-4844-8a60-1976734dd5e9","Type":"ContainerStarted","Data":"30422d98b0e41af3ef5d4356c232cef1e3c7de009fd4de1dd6d7ad6ef0f7a11f"}
Oct 30 21:53:33.905360 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:33.905196    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:34.203120 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:34.203019    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps" podStartSLOduration=3.125838715 podStartE2EDuration="8.203003934s" totalImagesPullingTime="4.585594223s" totalInitContainerRuntime="491.570996ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:34.152172107 +0000 UTC m=+225.676314882" watchObservedRunningTime="2026-10-30 21:53:34.203003934 +0000 UTC m=+225.727146697"
Oct 30 21:53:36.430737 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:36.430696    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:36.663404 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:36.663373    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.718343 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:39.718312    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:44.616432 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:44.616399    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:44.912274 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:44.912182    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-gchps"
Oct 30 21:53:45.857604 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:45.857569    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:46.102508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:46.102447    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:48.808392 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:48.808352    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:50.099031 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099003    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"]
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099365    5031 cpu_manager.go:414] "containerMap: removing container" podUID="59f70b6d-06f2-4890-995e-e1df707e9389" containerName="pull"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099379    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f70b6d-06f2-4890-995e-e1df707e9389" containerName="pull"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099390    5031 cpu_manager.go:414] "containerMap: removing container" podUID="361999d0-8e99-4235-be6f-9702118856a4" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099395    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361999d0-8e99-4235-be6f-9702118856a4" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099404    5031 cpu_manager.go:414] "containerMap: removing container" podUID="59f70b6d-06f2-4890-995e-e1df707e9389" containerName="util"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099409    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f70b6d-06f2-4890-995e-e1df707e9389" containerName="util"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099414    5031 cpu_manager.go:414] "containerMap: removing container" podUID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099418    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099426    5031 cpu_manager.go:414] "containerMap: removing container" podUID="361999d0-8e99-4235-be6f-9702118856a4" containerName="util"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099431    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361999d0-8e99-4235-be6f-9702118856a4" containerName="util"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099438    5031 cpu_manager.go:414] "containerMap: removing container" podUID="700db79a-2646-4811-b810-12df4108dfec" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099442    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700db79a-2646-4811-b810-12df4108dfec" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099449    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerName="pull"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099454    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerName="pull"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099461    5031 cpu_manager.go:414] "containerMap: removing container" podUID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099465    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerName="extract"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099473    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerName="util"
Oct 30 21:53:50.099471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099477    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099483    5031 cpu_manager.go:414] "containerMap: removing container" podUID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099487    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099498    5031 cpu_manager.go:414] "containerMap: removing container" podUID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099502    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099507    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099512    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099517    5031 cpu_manager.go:414] "containerMap: removing container" podUID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099520    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099532    5031 cpu_manager.go:414] "containerMap: removing container" podUID="700db79a-2646-4811-b810-12df4108dfec" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099536    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700db79a-2646-4811-b810-12df4108dfec" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099541    5031 cpu_manager.go:414] "containerMap: removing container" podUID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099545    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099550    5031 cpu_manager.go:414] "containerMap: removing container" podUID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099554    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099559    5031 cpu_manager.go:414] "containerMap: removing container" podUID="59f70b6d-06f2-4890-995e-e1df707e9389" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099563    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f70b6d-06f2-4890-995e-e1df707e9389" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099569    5031 cpu_manager.go:414] "containerMap: removing container" podUID="700db79a-2646-4811-b810-12df4108dfec" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099574    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700db79a-2646-4811-b810-12df4108dfec" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099585    5031 cpu_manager.go:414] "containerMap: removing container" podUID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099589    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099594    5031 cpu_manager.go:414] "containerMap: removing container" podUID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099598    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerName="util"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099603    5031 cpu_manager.go:414] "containerMap: removing container" podUID="361999d0-8e99-4235-be6f-9702118856a4" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099606    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361999d0-8e99-4235-be6f-9702118856a4" containerName="pull"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099658    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0572803-97c0-4ae7-805d-1ca9503a5385" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099667    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="279b4f64-0057-41ee-8b9b-5ebbef2b4aa5" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099672    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f70b6d-06f2-4890-995e-e1df707e9389" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099680    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="361999d0-8e99-4235-be6f-9702118856a4" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099686    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e9baa1-8db0-420a-9e51-26b5a738eaff" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099691    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="700db79a-2646-4811-b810-12df4108dfec" containerName="extract"
Oct 30 21:53:50.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.099697    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="415cebd3-87d4-49aa-8e02-27fcc02b0b29" containerName="extract"
Oct 30 21:53:50.103707 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.103687    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"
Oct 30 21:53:50.167644 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.167599    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9vt76\""
Oct 30 21:53:50.168865 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.168842    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:53:50.168865 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.168843    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:53:50.192361 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.192318    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ksj2\" (UniqueName: \"kubernetes.io/projected/5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e-kube-api-access-4ksj2\") pod \"authorino-operator-7b7bb4bc77-dmnft\" (UID: \"5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"
Oct 30 21:53:50.221774 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.221747    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"]
Oct 30 21:53:50.293713 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.293681    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ksj2\" (UniqueName: \"kubernetes.io/projected/5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e-kube-api-access-4ksj2\") pod \"authorino-operator-7b7bb4bc77-dmnft\" (UID: \"5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"
Oct 30 21:53:50.357306 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.357230    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ksj2\" (UniqueName: \"kubernetes.io/projected/5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e-kube-api-access-4ksj2\") pod \"authorino-operator-7b7bb4bc77-dmnft\" (UID: \"5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"
Oct 30 21:53:50.415309 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.415271    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"
Oct 30 21:53:50.539829 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:53:50.539789    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:53:50.791689 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.791662    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"]
Oct 30 21:53:50.793845 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:53:50.793802    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb8d3ea_7ffc_4afe_9e2a_f549e3731b8e.slice/crio-e6b2f9d06ab80d942c2db213aaff3ca6963caf805341143256cc88f7b2c4ac63 WatchSource:0}: Error finding container e6b2f9d06ab80d942c2db213aaff3ca6963caf805341143256cc88f7b2c4ac63: Status 404 returned error can't find the container with id e6b2f9d06ab80d942c2db213aaff3ca6963caf805341143256cc88f7b2c4ac63
Oct 30 21:53:50.984145 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:50.984110    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft" event={"ID":"5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e","Type":"ContainerStarted","Data":"e6b2f9d06ab80d942c2db213aaff3ca6963caf805341143256cc88f7b2c4ac63"}
Oct 30 21:53:51.233849 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:51.233811    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:51.643057 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:51.643025    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:51.901707 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:51.901628    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:52.145077 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.145043    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:52.565028 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.564989    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"]
Oct 30 21:53:52.567612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.567595    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:52.619429 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.619397    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"]
Oct 30 21:53:52.667838 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.667806    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 21:53:52.667838 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.667806    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:53:52.668072 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.667931    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-cwwb8\""
Oct 30 21:53:52.711896 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.711862    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b358167-be9d-4f81-9b14-23a009b17b68-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4\" (UID: \"4b358167-be9d-4f81-9b14-23a009b17b68\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:52.712067 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.711927    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89kh\" (UniqueName: \"kubernetes.io/projected/4b358167-be9d-4f81-9b14-23a009b17b68-kube-api-access-n89kh\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4\" (UID: \"4b358167-be9d-4f81-9b14-23a009b17b68\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:52.813035 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.813000    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89kh\" (UniqueName: \"kubernetes.io/projected/4b358167-be9d-4f81-9b14-23a009b17b68-kube-api-access-n89kh\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4\" (UID: \"4b358167-be9d-4f81-9b14-23a009b17b68\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:52.813252 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.813056    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b358167-be9d-4f81-9b14-23a009b17b68-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4\" (UID: \"4b358167-be9d-4f81-9b14-23a009b17b68\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:52.813484 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.813468    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b358167-be9d-4f81-9b14-23a009b17b68-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4\" (UID: \"4b358167-be9d-4f81-9b14-23a009b17b68\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:52.879697 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:52.879664    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89kh\" (UniqueName: \"kubernetes.io/projected/4b358167-be9d-4f81-9b14-23a009b17b68-kube-api-access-n89kh\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4\" (UID: \"4b358167-be9d-4f81-9b14-23a009b17b68\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:53.046649 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:53.046614    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:53.178923 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:53.178840    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"
Oct 30 21:53:53.655933 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:53.655909    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4"]
Oct 30 21:53:53.658333 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:53:53.658294    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b358167_be9d_4f81_9b14_23a009b17b68.slice/crio-82058fa3f546c689e4ef8c5adef4c6863cdc34e8ef8177b12dcb1344592d05b7 WatchSource:0}: Error finding container 82058fa3f546c689e4ef8c5adef4c6863cdc34e8ef8177b12dcb1344592d05b7: Status 404 returned error can't find the container with id 82058fa3f546c689e4ef8c5adef4c6863cdc34e8ef8177b12dcb1344592d05b7
Oct 30 21:53:53.999650 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:53.999572    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4" event={"ID":"4b358167-be9d-4f81-9b14-23a009b17b68","Type":"ContainerStarted","Data":"82058fa3f546c689e4ef8c5adef4c6863cdc34e8ef8177b12dcb1344592d05b7"}
Oct 30 21:53:55.290514 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:55.290455    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:55.626261 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:55.626225    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:56.461464 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:56.461409    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:56.711644 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:56.711531    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:56.945428 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:56.945378    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:57.015458 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.015378    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4" event={"ID":"4b358167-be9d-4f81-9b14-23a009b17b68","Type":"ContainerStarted","Data":"7062d825f9c26ad00b029a4e744c169510bfdb9363ec085398c7c1f0a05d5fac"}
Oct 30 21:53:57.241435 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.241376    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lc9j4" podStartSLOduration=2.35613933 podStartE2EDuration="5.241360019s" totalImagesPullingTime="2.885220689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:57.240398904 +0000 UTC m=+248.764541669" watchObservedRunningTime="2026-10-30 21:53:57.241360019 +0000 UTC m=+248.765502782"
Oct 30 21:53:57.607790 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.607749    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"]
Oct 30 21:53:57.611143 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.611127    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:57.665824 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.665791    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z97mv\""
Oct 30 21:53:57.717784 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.717748    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"]
Oct 30 21:53:57.757163 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.757120    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhp6p\" (UniqueName: \"kubernetes.io/projected/22249053-b464-4921-bf38-544444b708c7-kube-api-access-xhp6p\") pod \"kuadrant-operator-controller-manager-678d87766f-bhrv6\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:57.757163 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.757170    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22249053-b464-4921-bf38-544444b708c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bhrv6\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:57.858537 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.858447    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhp6p\" (UniqueName: \"kubernetes.io/projected/22249053-b464-4921-bf38-544444b708c7-kube-api-access-xhp6p\") pod \"kuadrant-operator-controller-manager-678d87766f-bhrv6\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:57.858537 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.858495    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22249053-b464-4921-bf38-544444b708c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bhrv6\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:57.858833 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.858817    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22249053-b464-4921-bf38-544444b708c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bhrv6\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:57.937378 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:57.937347    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhp6p\" (UniqueName: \"kubernetes.io/projected/22249053-b464-4921-bf38-544444b708c7-kube-api-access-xhp6p\") pod \"kuadrant-operator-controller-manager-678d87766f-bhrv6\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:58.230262 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:58.230166    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:53:58.505862 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:58.505752    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:58.722044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:58.721995    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"]
Oct 30 21:53:58.756766 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:58.756683    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:58.774112 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:53:58.774079    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22249053_b464_4921_bf38_544444b708c7.slice/crio-eea89a4b1887d6a795924e2ee12728700fdeb6fedcf58f2bce292ccdd83f63ae WatchSource:0}: Error finding container eea89a4b1887d6a795924e2ee12728700fdeb6fedcf58f2bce292ccdd83f63ae: Status 404 returned error can't find the container with id eea89a4b1887d6a795924e2ee12728700fdeb6fedcf58f2bce292ccdd83f63ae
Oct 30 21:53:59.025594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:59.025503    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6" event={"ID":"22249053-b464-4921-bf38-544444b708c7","Type":"ContainerStarted","Data":"eea89a4b1887d6a795924e2ee12728700fdeb6fedcf58f2bce292ccdd83f63ae"}
Oct 30 21:53:59.434749 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:53:59.434715    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:00.336962 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:00.336924    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:01.195565 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.195533    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:01.524665 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.524580    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"]
Oct 30 21:54:01.528648 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.528617    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"
Oct 30 21:54:01.591088 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.591005    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"]
Oct 30 21:54:01.638500 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.638466    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 21:54:01.656636 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.656603    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q52tg\""
Oct 30 21:54:01.695004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.694763    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhgm\" (UniqueName: \"kubernetes.io/projected/b583470d-0cc1-4196-8225-4ffcb972f282-kube-api-access-cqhgm\") pod \"dns-operator-controller-manager-8557cbdb84-ms8lr\" (UID: \"b583470d-0cc1-4196-8225-4ffcb972f282\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"
Oct 30 21:54:01.796419 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.796385    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhgm\" (UniqueName: \"kubernetes.io/projected/b583470d-0cc1-4196-8225-4ffcb972f282-kube-api-access-cqhgm\") pod \"dns-operator-controller-manager-8557cbdb84-ms8lr\" (UID: \"b583470d-0cc1-4196-8225-4ffcb972f282\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"
Oct 30 21:54:01.866292 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:01.866266    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhgm\" (UniqueName: \"kubernetes.io/projected/b583470d-0cc1-4196-8225-4ffcb972f282-kube-api-access-cqhgm\") pod \"dns-operator-controller-manager-8557cbdb84-ms8lr\" (UID: \"b583470d-0cc1-4196-8225-4ffcb972f282\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"
Oct 30 21:54:02.045145 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:02.044755    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft" event={"ID":"5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e","Type":"ContainerStarted","Data":"a0b9fa8affe0d11499a4e12e64dcdde61a9628ea31499976a936f03db1829e41"}
Oct 30 21:54:02.045145 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:02.044838    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"
Oct 30 21:54:02.146726 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:02.146687    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"
Oct 30 21:54:02.179067 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:02.179007    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft" podStartSLOduration=1.229181561 podStartE2EDuration="12.178987106s" totalImagesPullingTime="10.949805545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:02.178307891 +0000 UTC m=+253.702450655" watchObservedRunningTime="2026-10-30 21:54:02.178987106 +0000 UTC m=+253.703129869"
Oct 30 21:54:02.540146 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:02.540113    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"]
Oct 30 21:54:02.602193 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:02.602156    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:02.836679 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:02.836600    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:03.052917 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:03.052873    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr" event={"ID":"b583470d-0cc1-4196-8225-4ffcb972f282","Type":"ContainerStarted","Data":"5967bd26060b41f0819a9a801dece4c036649a247152a1dc0fd96f14088d6357"}
Oct 30 21:54:03.601391 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:03.601336    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:03.873463 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:03.873389    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:04.239581 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:04.239491    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:04.669502 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:04.669470    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:04.913914 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:04.913882    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:05.875874 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:05.875842    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:05.949058 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:05.949028    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"]
Oct 30 21:54:05.952094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:05.952069    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"
Oct 30 21:54:06.024161 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.024127    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"]
Oct 30 21:54:06.029585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.029523    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6gnzq\""
Oct 30 21:54:06.034975 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.034946    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4zgs\" (UniqueName: \"kubernetes.io/projected/51231b17-e61d-414b-a7d9-3ef076e2fcf5-kube-api-access-r4zgs\") pod \"limitador-operator-controller-manager-5778bb4449-cw5tc\" (UID: \"51231b17-e61d-414b-a7d9-3ef076e2fcf5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"
Oct 30 21:54:06.071080 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.071042    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6" event={"ID":"22249053-b464-4921-bf38-544444b708c7","Type":"ContainerStarted","Data":"0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325"}
Oct 30 21:54:06.071265 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.071103    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:54:06.073264 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.073235    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr" event={"ID":"b583470d-0cc1-4196-8225-4ffcb972f282","Type":"ContainerStarted","Data":"6d5fa375fcdb4fb00071a3f7533e7611c00268460e7a5e5a4ba6c5437431d5d3"}
Oct 30 21:54:06.073367 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.073350    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"
Oct 30 21:54:06.127401 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.127372    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:06.136560 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.136531    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4zgs\" (UniqueName: \"kubernetes.io/projected/51231b17-e61d-414b-a7d9-3ef076e2fcf5-kube-api-access-r4zgs\") pod \"limitador-operator-controller-manager-5778bb4449-cw5tc\" (UID: \"51231b17-e61d-414b-a7d9-3ef076e2fcf5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"
Oct 30 21:54:06.197861 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.197814    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6" podStartSLOduration=2.193795363 podStartE2EDuration="9.197799422s" totalImagesPullingTime="7.004004059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:06.195345774 +0000 UTC m=+257.719488564" watchObservedRunningTime="2026-10-30 21:54:06.197799422 +0000 UTC m=+257.721942186"
Oct 30 21:54:06.202360 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.202334    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4zgs\" (UniqueName: \"kubernetes.io/projected/51231b17-e61d-414b-a7d9-3ef076e2fcf5-kube-api-access-r4zgs\") pod \"limitador-operator-controller-manager-5778bb4449-cw5tc\" (UID: \"51231b17-e61d-414b-a7d9-3ef076e2fcf5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"
Oct 30 21:54:06.266232 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.265605    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"
Oct 30 21:54:06.320023 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.318822    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr" podStartSLOduration=2.451176251 podStartE2EDuration="5.318802835s" totalImagesPullingTime="2.867626584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:06.317631415 +0000 UTC m=+257.841774176" watchObservedRunningTime="2026-10-30 21:54:06.318802835 +0000 UTC m=+257.842945599"
Oct 30 21:54:06.353385 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.353346    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:06.541833 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.541625    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"]
Oct 30 21:54:06.561015 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.546554    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.610094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.610063    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"]
Oct 30 21:54:06.641135 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641105    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641135 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641144    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641175    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed454-d15a-4895-8a38-bef8ac996847-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641307    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqcfk\" (UniqueName: \"kubernetes.io/projected/eaeed454-d15a-4895-8a38-bef8ac996847-kube-api-access-sqcfk\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641368    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641394    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed454-d15a-4895-8a38-bef8ac996847-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641482    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed454-d15a-4895-8a38-bef8ac996847-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641518    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.641674 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.641599    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.646024 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.646003    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7nbj5\""
Oct 30 21:54:06.655336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.655319    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"]
Oct 30 21:54:06.657483 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:54:06.657455    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51231b17_e61d_414b_a7d9_3ef076e2fcf5.slice/crio-282a025b1e8e62d59c4a6d00f9643c33959de7b19b915e392044c4d9330a6b01 WatchSource:0}: Error finding container 282a025b1e8e62d59c4a6d00f9643c33959de7b19b915e392044c4d9330a6b01: Status 404 returned error can't find the container with id 282a025b1e8e62d59c4a6d00f9643c33959de7b19b915e392044c4d9330a6b01
Oct 30 21:54:06.742955 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.742922    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed454-d15a-4895-8a38-bef8ac996847-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.742955 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.742964    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqcfk\" (UniqueName: \"kubernetes.io/projected/eaeed454-d15a-4895-8a38-bef8ac996847-kube-api-access-sqcfk\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743014    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743033    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed454-d15a-4895-8a38-bef8ac996847-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743058    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed454-d15a-4895-8a38-bef8ac996847-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743117    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743166    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743221    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743550 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743276    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743596 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743561    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743657 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743633    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743780 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743751    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.743894 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.743870    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.744317 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.744282    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeed454-d15a-4895-8a38-bef8ac996847-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.745577 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.745548    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:06.745894 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.745867    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeed454-d15a-4895-8a38-bef8ac996847-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.746191 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.746172    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeed454-d15a-4895-8a38-bef8ac996847-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.807127 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.807048    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqcfk\" (UniqueName: \"kubernetes.io/projected/eaeed454-d15a-4895-8a38-bef8ac996847-kube-api-access-sqcfk\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.812142 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.810203    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeed454-d15a-4895-8a38-bef8ac996847-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8\" (UID: \"eaeed454-d15a-4895-8a38-bef8ac996847\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:06.860458 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:06.860422    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:07.076270 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:07.076166    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:07.080351 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:07.080316    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc" event={"ID":"51231b17-e61d-414b-a7d9-3ef076e2fcf5","Type":"ContainerStarted","Data":"282a025b1e8e62d59c4a6d00f9643c33959de7b19b915e392044c4d9330a6b01"}
Oct 30 21:54:07.290344 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:07.290303    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"]
Oct 30 21:54:07.294636 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:54:07.294606    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaeed454_d15a_4895_8a38_bef8ac996847.slice/crio-70481fd2074a96c6a5d23cadefbe5bb9c5097ae61cce5eedd3a664e06f0eae1b WatchSource:0}: Error finding container 70481fd2074a96c6a5d23cadefbe5bb9c5097ae61cce5eedd3a664e06f0eae1b: Status 404 returned error can't find the container with id 70481fd2074a96c6a5d23cadefbe5bb9c5097ae61cce5eedd3a664e06f0eae1b
Oct 30 21:54:07.296815 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:07.296781    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:54:07.296899 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:07.296849    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:54:07.296899 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:07.296880    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:54:07.824044 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:07.823994    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:08.066726 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:08.066684    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:08.087008 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:08.086924    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8" event={"ID":"eaeed454-d15a-4895-8a38-bef8ac996847","Type":"ContainerStarted","Data":"8782b9f5215b647c6d4677c73fb56e8a5ba993dadee0e8090d2b9c047388f750"}
Oct 30 21:54:08.087008 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:08.086988    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8" event={"ID":"eaeed454-d15a-4895-8a38-bef8ac996847","Type":"ContainerStarted","Data":"70481fd2074a96c6a5d23cadefbe5bb9c5097ae61cce5eedd3a664e06f0eae1b"}
Oct 30 21:54:08.216664 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:08.216610    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8" podStartSLOduration=2.216597055 podStartE2EDuration="2.216597055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:08.215941818 +0000 UTC m=+259.740084582" watchObservedRunningTime="2026-10-30 21:54:08.216597055 +0000 UTC m=+259.740739819"
Oct 30 21:54:08.288171 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:08.288134    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:08.861027 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:08.860996    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:08.868032 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:08.868008    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:09.093734 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:09.093707    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:09.095122 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:09.095102    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8"
Oct 30 21:54:10.100182 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:10.100150    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc" event={"ID":"51231b17-e61d-414b-a7d9-3ef076e2fcf5","Type":"ContainerStarted","Data":"1c13ac369c1369a92e8c19f16befe6a715db6ab89389e071ba324e7bb5974672"}
Oct 30 21:54:10.100581 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:10.100364    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"
Oct 30 21:54:10.120567 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:10.120542    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.221395 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:10.221346    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc" podStartSLOduration=2.55589838 podStartE2EDuration="5.221331807s" totalImagesPullingTime="2.665433427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.219986923 +0000 UTC m=+261.744129687" watchObservedRunningTime="2026-10-30 21:54:10.221331807 +0000 UTC m=+261.745474571"
Oct 30 21:54:10.347190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:10.347150    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.604283 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:10.604252    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.831070 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:10.831036    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:13.054904 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.054873    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dmnft"
Oct 30 21:54:13.163275 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.163245    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8k2z"]
Oct 30 21:54:13.168255 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.168051    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:54:13.168403 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.168270    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z"
Oct 30 21:54:13.171382 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.171361    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.206789 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.206756    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98377d7d-c750-4134-8611-00d9103d85b0-config-file\") pod \"limitador-limitador-7c9b87d574-j8xxp\" (UID: \"98377d7d-c750-4134-8611-00d9103d85b0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.206945 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.206855    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdrr\" (UniqueName: \"kubernetes.io/projected/98377d7d-c750-4134-8611-00d9103d85b0-kube-api-access-gjdrr\") pod \"limitador-limitador-7c9b87d574-j8xxp\" (UID: \"98377d7d-c750-4134-8611-00d9103d85b0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.206945 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.206912    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2f5x\" (UniqueName: \"kubernetes.io/projected/a99bc674-28d0-4503-8cc4-1334e1ed69aa-kube-api-access-w2f5x\") pod \"authorino-56c6c9ddd5-w8k2z\" (UID: \"a99bc674-28d0-4503-8cc4-1334e1ed69aa\") " pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z"
Oct 30 21:54:13.215408 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.215379    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8k2z"]
Oct 30 21:54:13.231246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.231202    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z7pdc\""
Oct 30 21:54:13.231246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.231234    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 21:54:13.231246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.231202    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-86hk6\""
Oct 30 21:54:13.252572 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.252545    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:54:13.308736 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.308658    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdrr\" (UniqueName: \"kubernetes.io/projected/98377d7d-c750-4134-8611-00d9103d85b0-kube-api-access-gjdrr\") pod \"limitador-limitador-7c9b87d574-j8xxp\" (UID: \"98377d7d-c750-4134-8611-00d9103d85b0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.308736 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.308728    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f5x\" (UniqueName: \"kubernetes.io/projected/a99bc674-28d0-4503-8cc4-1334e1ed69aa-kube-api-access-w2f5x\") pod \"authorino-56c6c9ddd5-w8k2z\" (UID: \"a99bc674-28d0-4503-8cc4-1334e1ed69aa\") " pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z"
Oct 30 21:54:13.308907 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.308810    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98377d7d-c750-4134-8611-00d9103d85b0-config-file\") pod \"limitador-limitador-7c9b87d574-j8xxp\" (UID: \"98377d7d-c750-4134-8611-00d9103d85b0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.309556 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.309536    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98377d7d-c750-4134-8611-00d9103d85b0-config-file\") pod \"limitador-limitador-7c9b87d574-j8xxp\" (UID: \"98377d7d-c750-4134-8611-00d9103d85b0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.339220 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.339190    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:13.377899 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.377872    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdrr\" (UniqueName: \"kubernetes.io/projected/98377d7d-c750-4134-8611-00d9103d85b0-kube-api-access-gjdrr\") pod \"limitador-limitador-7c9b87d574-j8xxp\" (UID: \"98377d7d-c750-4134-8611-00d9103d85b0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.378026 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.377981    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2f5x\" (UniqueName: \"kubernetes.io/projected/a99bc674-28d0-4503-8cc4-1334e1ed69aa-kube-api-access-w2f5x\") pod \"authorino-56c6c9ddd5-w8k2z\" (UID: \"a99bc674-28d0-4503-8cc4-1334e1ed69aa\") " pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z"
Oct 30 21:54:13.448498 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.448466    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:54:13.490323 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.490283    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z"
Oct 30 21:54:13.499129 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.499105    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:13.701181 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.701149    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:13.880831 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.880811    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8k2z"]
Oct 30 21:54:13.882187 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:54:13.882165    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda99bc674_28d0_4503_8cc4_1334e1ed69aa.slice/crio-2f3e6e3243c13a9f53d292d246015311796aeacb5412d9ff715a92f1713f75bc WatchSource:0}: Error finding container 2f3e6e3243c13a9f53d292d246015311796aeacb5412d9ff715a92f1713f75bc: Status 404 returned error can't find the container with id 2f3e6e3243c13a9f53d292d246015311796aeacb5412d9ff715a92f1713f75bc
Oct 30 21:54:13.931693 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.931662    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:13.933736 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:13.933712    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:54:13.934972 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:54:13.934951    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98377d7d_c750_4134_8611_00d9103d85b0.slice/crio-7b6a9c8556c1e58c67da1c5531d11319a98ef985639a55306e4150db9e607aeb WatchSource:0}: Error finding container 7b6a9c8556c1e58c67da1c5531d11319a98ef985639a55306e4150db9e607aeb: Status 404 returned error can't find the container with id 7b6a9c8556c1e58c67da1c5531d11319a98ef985639a55306e4150db9e607aeb
Oct 30 21:54:14.120356 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:14.120323    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z" event={"ID":"a99bc674-28d0-4503-8cc4-1334e1ed69aa","Type":"ContainerStarted","Data":"2f3e6e3243c13a9f53d292d246015311796aeacb5412d9ff715a92f1713f75bc"}
Oct 30 21:54:14.121665 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:14.121634    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp" event={"ID":"98377d7d-c750-4134-8611-00d9103d85b0","Type":"ContainerStarted","Data":"7b6a9c8556c1e58c67da1c5531d11319a98ef985639a55306e4150db9e607aeb"}
Oct 30 21:54:16.022591 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:16.022510    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:16.143246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:16.143159    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp" event={"ID":"98377d7d-c750-4134-8611-00d9103d85b0","Type":"ContainerStarted","Data":"b3eecf7ce87b906b57842a92a18f8635e391768c5ea70c931d274944c13ed7c8"}
Oct 30 21:54:16.143399 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:16.143337    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:16.274781 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:16.274740    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp" podStartSLOduration=1.3325546799999999 podStartE2EDuration="3.274726637s" totalImagesPullingTime="1.942171957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:16.273308399 +0000 UTC m=+267.797451161" watchObservedRunningTime="2026-10-30 21:54:16.274726637 +0000 UTC m=+267.798869400"
Oct 30 21:54:16.280503 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:16.280478    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:16.625109 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:16.625081    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:16.883133 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:16.883054    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:17.082569 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:17.082542    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 21:54:17.083003 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:17.082620    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ms8lr"
Oct 30 21:54:17.134448 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:17.134369    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:17.150383 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:17.150357    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z" event={"ID":"a99bc674-28d0-4503-8cc4-1334e1ed69aa","Type":"ContainerStarted","Data":"9e73ef7be20e1b8cabee06c414d90b70aca3ccfffdb3253602bfcd67dc687db7"}
Oct 30 21:54:17.483751 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:17.483661    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z" podStartSLOduration=2.160368371 podStartE2EDuration="4.483645946s" totalImagesPullingTime="2.323277575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:17.481974975 +0000 UTC m=+269.006117738" watchObservedRunningTime="2026-10-30 21:54:17.483645946 +0000 UTC m=+269.007788708"
Oct 30 21:54:20.540306 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:54:20.540274    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:54:21.108193 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:21.108163    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cw5tc"
Oct 30 21:54:21.735805 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:21.735769    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:21.963789 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:21.963745    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:26.468162 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:26.468125    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:26.723276 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:26.723189    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:27.151768 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:27.151743    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"
Oct 30 21:54:29.406563 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:29.406505    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:29.711202 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:29.711127    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:31.468168 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:31.468138    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:31.687412 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:31.687381    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:32.243691 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:32.243645    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:35.207634 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:35.207570    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:35.425203 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:35.425166    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:35.645841 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:35.645804    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:36.016591 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:36.016506    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:36.237895 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:36.237859    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:40.074617 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:40.074586    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:40.358219 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:40.358181    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:42.805613 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:42.805584    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:43.030005 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:43.029966    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:46.622072 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:46.622034    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:46.851881 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:46.851847    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:48.658822 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:48.658793    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:54:48.659858 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:48.659837    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:54:50.539827 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:54:50.539790    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:54:51.928088 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:51.928051    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:52.151986 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:52.151949    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:54.729849 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:54.729817    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:54.953409 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:54.953376    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:55.181585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:55.181553    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:59.664272 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:59.664177    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:59.894134 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:54:59.894102    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:02.802350 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:02.802317    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:03.197797 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:03.197763    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:20.539424 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:55:20.539377    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:55:32.951692 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:32.951647    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:33.188071 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:33.188042    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:35.158079 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:35.158046    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:35.391038 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:35.390997    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:35.621914 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:35.621879    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:40.425483 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:40.425442    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:40.649390 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:40.649359    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:55:41.597450 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:41.597412    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:41.839520 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:41.839485    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:42.079567 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.079494    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:42.331547 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.331459    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:42.557918 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.557885    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:42.651029 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.651003    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-92hjx"]
Oct 30 21:55:42.654690 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.654670    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.731937 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.722154    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-92hjx"]
Oct 30 21:55:42.769158 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.769125    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:55:42.769158 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.769142    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:55:42.769395 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.769238    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 21:55:42.769500 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.769481    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s6fk9\""
Oct 30 21:55:42.771049 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.771028    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-9fbbk"]
Oct 30 21:55:42.774796 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.774781    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.792455 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.792434    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrlg6\" (UniqueName: \"kubernetes.io/projected/f5932308-a52b-42d0-9706-2776b7e7fea4-kube-api-access-jrlg6\") pod \"model-serving-api-75bd89fdfc-92hjx\" (UID: \"f5932308-a52b-42d0-9706-2776b7e7fea4\") " pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.792558 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.792479    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5932308-a52b-42d0-9706-2776b7e7fea4-tls-certs\") pod \"model-serving-api-75bd89fdfc-92hjx\" (UID: \"f5932308-a52b-42d0-9706-2776b7e7fea4\") " pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.831486 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.831455    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:42.831720 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.831697    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 21:55:42.831884 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.831606    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 21:55:42.831973 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.831831    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bbdld\""
Oct 30 21:55:42.835091 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.835064    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9fbbk"]
Oct 30 21:55:42.892949 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.892925    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-metrics-certs\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.893067 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.892953    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qz2\" (UniqueName: \"kubernetes.io/projected/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-kube-api-access-t9qz2\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.893067 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.892978    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5932308-a52b-42d0-9706-2776b7e7fea4-tls-certs\") pod \"model-serving-api-75bd89fdfc-92hjx\" (UID: \"f5932308-a52b-42d0-9706-2776b7e7fea4\") " pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.893167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.893124    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrlg6\" (UniqueName: \"kubernetes.io/projected/f5932308-a52b-42d0-9706-2776b7e7fea4-kube-api-access-jrlg6\") pod \"model-serving-api-75bd89fdfc-92hjx\" (UID: \"f5932308-a52b-42d0-9706-2776b7e7fea4\") " pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.893167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.893143    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-cert\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.895758 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.895734    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5932308-a52b-42d0-9706-2776b7e7fea4-tls-certs\") pod \"model-serving-api-75bd89fdfc-92hjx\" (UID: \"f5932308-a52b-42d0-9706-2776b7e7fea4\") " pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.957903 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.957834    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrlg6\" (UniqueName: \"kubernetes.io/projected/f5932308-a52b-42d0-9706-2776b7e7fea4-kube-api-access-jrlg6\") pod \"model-serving-api-75bd89fdfc-92hjx\" (UID: \"f5932308-a52b-42d0-9706-2776b7e7fea4\") " pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.966711 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.966684    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:42.993928 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.993904    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-cert\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.994064 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.993948    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-metrics-certs\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.994064 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.993973    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qz2\" (UniqueName: \"kubernetes.io/projected/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-kube-api-access-t9qz2\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.996427 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.996403    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-cert\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:42.996612 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:42.996597    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-metrics-certs\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:43.061004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.060976    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qz2\" (UniqueName: \"kubernetes.io/projected/6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1-kube-api-access-t9qz2\") pod \"odh-model-controller-769875db56-9fbbk\" (UID: \"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1\") " pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:43.086969 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.086940    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:43.368633 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.368610    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-92hjx"]
Oct 30 21:55:43.370807 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:55:43.370776    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5932308_a52b_42d0_9706_2776b7e7fea4.slice/crio-92db938a3957e943c1ada87620eddf5700a60aefc4d4c60465c1245f1285242e WatchSource:0}: Error finding container 92db938a3957e943c1ada87620eddf5700a60aefc4d4c60465c1245f1285242e: Status 404 returned error can't find the container with id 92db938a3957e943c1ada87620eddf5700a60aefc4d4c60465c1245f1285242e
Oct 30 21:55:43.372685 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.372666    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:55:43.490438 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.490411    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9fbbk"]
Oct 30 21:55:43.492441 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:55:43.492408    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6d4cef_2c1a_4f0f_bfe4_5405883c1cb1.slice/crio-1fa1cb8194066941bc3ba9ccc89d2c90048e7fede8cc7b33d468132d14159e99 WatchSource:0}: Error finding container 1fa1cb8194066941bc3ba9ccc89d2c90048e7fede8cc7b33d468132d14159e99: Status 404 returned error can't find the container with id 1fa1cb8194066941bc3ba9ccc89d2c90048e7fede8cc7b33d468132d14159e99
Oct 30 21:55:43.634043 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.633949    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9fbbk" event={"ID":"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1","Type":"ContainerStarted","Data":"1fa1cb8194066941bc3ba9ccc89d2c90048e7fede8cc7b33d468132d14159e99"}
Oct 30 21:55:43.635262 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.635232    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-92hjx" event={"ID":"f5932308-a52b-42d0-9706-2776b7e7fea4","Type":"ContainerStarted","Data":"92db938a3957e943c1ada87620eddf5700a60aefc4d4c60465c1245f1285242e"}
Oct 30 21:55:43.898872 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:43.898802    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:45.442564 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:45.442494    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:45.663075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:45.663033    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:45.688788 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:45.688755    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5hl8c"]
Oct 30 21:55:45.692955 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:45.692904    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c"
Oct 30 21:55:45.752280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:45.750261    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5hl8c"]
Oct 30 21:55:45.844946 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:45.844898    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvj75\" (UniqueName: \"kubernetes.io/projected/4a18789c-e0f6-4d05-b12b-34226d2c2ec8-kube-api-access-bvj75\") pod \"authorino-6f6944bcdd-5hl8c\" (UID: \"4a18789c-e0f6-4d05-b12b-34226d2c2ec8\") " pod="kuadrant-system/authorino-6f6944bcdd-5hl8c"
Oct 30 21:55:45.946664 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:45.946464    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvj75\" (UniqueName: \"kubernetes.io/projected/4a18789c-e0f6-4d05-b12b-34226d2c2ec8-kube-api-access-bvj75\") pod \"authorino-6f6944bcdd-5hl8c\" (UID: \"4a18789c-e0f6-4d05-b12b-34226d2c2ec8\") " pod="kuadrant-system/authorino-6f6944bcdd-5hl8c"
Oct 30 21:55:46.011609 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.011572    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvj75\" (UniqueName: \"kubernetes.io/projected/4a18789c-e0f6-4d05-b12b-34226d2c2ec8-kube-api-access-bvj75\") pod \"authorino-6f6944bcdd-5hl8c\" (UID: \"4a18789c-e0f6-4d05-b12b-34226d2c2ec8\") " pod="kuadrant-system/authorino-6f6944bcdd-5hl8c"
Oct 30 21:55:46.046597 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.046004    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5hl8c"]
Oct 30 21:55:46.046597 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.046419    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c"
Oct 30 21:55:46.079409 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.078697    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9db4c4959-dgbt8"]
Oct 30 21:55:46.085336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.085312    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9db4c4959-dgbt8"
Oct 30 21:55:46.146370 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.146305    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9db4c4959-dgbt8"]
Oct 30 21:55:46.249796 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.249650    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdz7\" (UniqueName: \"kubernetes.io/projected/fa7fa2e8-a4fe-4969-973d-19bad30e8533-kube-api-access-tgdz7\") pod \"authorino-9db4c4959-dgbt8\" (UID: \"fa7fa2e8-a4fe-4969-973d-19bad30e8533\") " pod="kuadrant-system/authorino-9db4c4959-dgbt8"
Oct 30 21:55:46.350530 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.350492    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdz7\" (UniqueName: \"kubernetes.io/projected/fa7fa2e8-a4fe-4969-973d-19bad30e8533-kube-api-access-tgdz7\") pod \"authorino-9db4c4959-dgbt8\" (UID: \"fa7fa2e8-a4fe-4969-973d-19bad30e8533\") " pod="kuadrant-system/authorino-9db4c4959-dgbt8"
Oct 30 21:55:46.414901 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.414863    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdz7\" (UniqueName: \"kubernetes.io/projected/fa7fa2e8-a4fe-4969-973d-19bad30e8533-kube-api-access-tgdz7\") pod \"authorino-9db4c4959-dgbt8\" (UID: \"fa7fa2e8-a4fe-4969-973d-19bad30e8533\") " pod="kuadrant-system/authorino-9db4c4959-dgbt8"
Oct 30 21:55:46.446798 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.446768    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5hl8c"]
Oct 30 21:55:46.448989 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:55:46.448962    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a18789c_e0f6_4d05_b12b_34226d2c2ec8.slice/crio-2877d7e4ae143504f9fc5da46657f37ba366ce71194007beae43e844f0cca0ce WatchSource:0}: Error finding container 2877d7e4ae143504f9fc5da46657f37ba366ce71194007beae43e844f0cca0ce: Status 404 returned error can't find the container with id 2877d7e4ae143504f9fc5da46657f37ba366ce71194007beae43e844f0cca0ce
Oct 30 21:55:46.512663 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.512587    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9db4c4959-dgbt8"]
Oct 30 21:55:46.512859 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.512842    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9db4c4959-dgbt8"
Oct 30 21:55:46.662751 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.662717    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c" event={"ID":"4a18789c-e0f6-4d05-b12b-34226d2c2ec8","Type":"ContainerStarted","Data":"2877d7e4ae143504f9fc5da46657f37ba366ce71194007beae43e844f0cca0ce"}
Oct 30 21:55:46.906644 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:46.906554    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9db4c4959-dgbt8"]
Oct 30 21:55:46.909140 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:55:46.909084    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa7fa2e8_a4fe_4969_973d_19bad30e8533.slice/crio-d874e45c0c6591c32e53f1e4da601c918f37b8f6d3a04d189f51abbc221ea9a8 WatchSource:0}: Error finding container d874e45c0c6591c32e53f1e4da601c918f37b8f6d3a04d189f51abbc221ea9a8: Status 404 returned error can't find the container with id d874e45c0c6591c32e53f1e4da601c918f37b8f6d3a04d189f51abbc221ea9a8
Oct 30 21:55:47.669457 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:47.669422    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9db4c4959-dgbt8" event={"ID":"fa7fa2e8-a4fe-4969-973d-19bad30e8533","Type":"ContainerStarted","Data":"d874e45c0c6591c32e53f1e4da601c918f37b8f6d3a04d189f51abbc221ea9a8"}
Oct 30 21:55:47.671248 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:47.671187    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9fbbk" event={"ID":"6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1","Type":"ContainerStarted","Data":"0fbe8d348075c6bf405b6457d3c77e159fdc3eda3780b116cd8bff392bf68339"}
Oct 30 21:55:47.671445 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:47.671420    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:47.672952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:47.672927    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-92hjx" event={"ID":"f5932308-a52b-42d0-9706-2776b7e7fea4","Type":"ContainerStarted","Data":"338cc2da1da228557afa81a4d35de6ae4d80becbd0decb5105325da0eec52400"}
Oct 30 21:55:47.673072 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:47.672994    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:55:47.794144 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:47.794050    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-9fbbk" podStartSLOduration=1.7929116 podStartE2EDuration="5.794035658s" totalImagesPullingTime="4.001124058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:47.792126126 +0000 UTC m=+359.316268905" watchObservedRunningTime="2026-10-30 21:55:47.794035658 +0000 UTC m=+359.318178417"
Oct 30 21:55:47.913515 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:47.913476    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-92hjx" podStartSLOduration=2.683263995 podStartE2EDuration="5.913464077s" totalImagesPullingTime="3.230200082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:47.912105753 +0000 UTC m=+359.436248515" watchObservedRunningTime="2026-10-30 21:55:47.913464077 +0000 UTC m=+359.437606840"
Oct 30 21:55:48.682452 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.682366    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9db4c4959-dgbt8" event={"ID":"fa7fa2e8-a4fe-4969-973d-19bad30e8533","Type":"ContainerStarted","Data":"14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631"}
Oct 30 21:55:48.682916 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.682393    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-9db4c4959-dgbt8" podUID="fa7fa2e8-a4fe-4969-973d-19bad30e8533" containerName="authorino" containerID="cri-o://14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631" gracePeriod=30
Oct 30 21:55:48.686255 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.685664    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c" event={"ID":"4a18789c-e0f6-4d05-b12b-34226d2c2ec8","Type":"ContainerStarted","Data":"e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739"}
Oct 30 21:55:48.686255 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.685693    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c" podUID="4a18789c-e0f6-4d05-b12b-34226d2c2ec8" containerName="authorino" containerID="cri-o://e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739" gracePeriod=30
Oct 30 21:55:48.800655 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.800609    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-9db4c4959-dgbt8" podStartSLOduration=1.365678706 podStartE2EDuration="2.800596189s" totalImagesPullingTime="1.434917483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:48.799963967 +0000 UTC m=+360.324106740" watchObservedRunningTime="2026-10-30 21:55:48.800596189 +0000 UTC m=+360.324738951"
Oct 30 21:55:48.924860 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.923905    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c" podStartSLOduration=2.520664734 podStartE2EDuration="3.923885839s" totalImagesPullingTime="1.403221105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:48.920740731 +0000 UTC m=+360.444883500" watchObservedRunningTime="2026-10-30 21:55:48.923885839 +0000 UTC m=+360.448028602"
Oct 30 21:55:48.981034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.980988    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c"
Oct 30 21:55:48.984870 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:48.984850    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9db4c4959-dgbt8"
Oct 30 21:55:49.078112 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.078084    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7fa2e8-a4fe-4969-973d-19bad30e8533-kube-api-access-tgdz7\" (UniqueName: \"kubernetes.io/projected/fa7fa2e8-a4fe-4969-973d-19bad30e8533-kube-api-access-tgdz7\") pod \"fa7fa2e8-a4fe-4969-973d-19bad30e8533\" (UID: \"fa7fa2e8-a4fe-4969-973d-19bad30e8533\") "
Oct 30 21:55:49.078302 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.078188    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a18789c-e0f6-4d05-b12b-34226d2c2ec8-kube-api-access-bvj75\" (UniqueName: \"kubernetes.io/projected/4a18789c-e0f6-4d05-b12b-34226d2c2ec8-kube-api-access-bvj75\") pod \"4a18789c-e0f6-4d05-b12b-34226d2c2ec8\" (UID: \"4a18789c-e0f6-4d05-b12b-34226d2c2ec8\") "
Oct 30 21:55:49.080377 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.080351    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7fa2e8-a4fe-4969-973d-19bad30e8533-kube-api-access-tgdz7" pod "fa7fa2e8-a4fe-4969-973d-19bad30e8533" (UID: "fa7fa2e8-a4fe-4969-973d-19bad30e8533"). InnerVolumeSpecName "kube-api-access-tgdz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:49.080377 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.080366    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a18789c-e0f6-4d05-b12b-34226d2c2ec8-kube-api-access-bvj75" pod "4a18789c-e0f6-4d05-b12b-34226d2c2ec8" (UID: "4a18789c-e0f6-4d05-b12b-34226d2c2ec8"). InnerVolumeSpecName "kube-api-access-bvj75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:49.171554 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.171516    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:49.179887 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.179856    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvj75\" (UniqueName: \"kubernetes.io/projected/4a18789c-e0f6-4d05-b12b-34226d2c2ec8-kube-api-access-bvj75\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:49.180011 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.179890    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdz7\" (UniqueName: \"kubernetes.io/projected/fa7fa2e8-a4fe-4969-973d-19bad30e8533-kube-api-access-tgdz7\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:49.500268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.500232    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:49.515760 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.515307    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8k2z"]
Oct 30 21:55:49.515760 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.515505    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z" podUID="a99bc674-28d0-4503-8cc4-1334e1ed69aa" containerName="authorino" containerID="cri-o://9e73ef7be20e1b8cabee06c414d90b70aca3ccfffdb3253602bfcd67dc687db7" gracePeriod=30
Oct 30 21:55:49.693382 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.693343    5031 generic.go:356] "Generic (PLEG): container finished" podID="de2775d7-88bd-440d-8034-5a7412a22128" containerID="b91146d0fda3ea260fbc46d0c6b6c464e1f3d648e767627ed2a0531c1e35fc0b" exitCode=0
Oct 30 21:55:49.693841 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.693410    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" event={"ID":"de2775d7-88bd-440d-8034-5a7412a22128","Type":"ContainerDied","Data":"b91146d0fda3ea260fbc46d0c6b6c464e1f3d648e767627ed2a0531c1e35fc0b"}
Oct 30 21:55:49.693935 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.693853    5031 scope.go:122] "RemoveContainer" containerID="b91146d0fda3ea260fbc46d0c6b6c464e1f3d648e767627ed2a0531c1e35fc0b"
Oct 30 21:55:49.695134 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695107    5031 generic.go:356] "Generic (PLEG): container finished" podID="4a18789c-e0f6-4d05-b12b-34226d2c2ec8" containerID="e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739" exitCode=0
Oct 30 21:55:49.695295 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695142    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c"
Oct 30 21:55:49.695364 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695278    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:55:49.695364 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695329    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c" event={"ID":"4a18789c-e0f6-4d05-b12b-34226d2c2ec8","Type":"ContainerDied","Data":"e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739"}
Oct 30 21:55:49.695467 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695362    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-5hl8c" event={"ID":"4a18789c-e0f6-4d05-b12b-34226d2c2ec8","Type":"ContainerDied","Data":"2877d7e4ae143504f9fc5da46657f37ba366ce71194007beae43e844f0cca0ce"}
Oct 30 21:55:49.695467 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695384    5031 scope.go:122] "RemoveContainer" containerID="e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739"
Oct 30 21:55:49.695780 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695570    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:55:49.695905 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.695780    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:55:49.700375 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.697375    5031 generic.go:356] "Generic (PLEG): container finished" podID="a99bc674-28d0-4503-8cc4-1334e1ed69aa" containerID="9e73ef7be20e1b8cabee06c414d90b70aca3ccfffdb3253602bfcd67dc687db7" exitCode=0
Oct 30 21:55:49.700375 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.697453    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z" event={"ID":"a99bc674-28d0-4503-8cc4-1334e1ed69aa","Type":"ContainerDied","Data":"9e73ef7be20e1b8cabee06c414d90b70aca3ccfffdb3253602bfcd67dc687db7"}
Oct 30 21:55:49.702119 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.702095    5031 generic.go:356] "Generic (PLEG): container finished" podID="fa7fa2e8-a4fe-4969-973d-19bad30e8533" containerID="14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631" exitCode=0
Oct 30 21:55:49.702258 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.702144    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9db4c4959-dgbt8" event={"ID":"fa7fa2e8-a4fe-4969-973d-19bad30e8533","Type":"ContainerDied","Data":"14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631"}
Oct 30 21:55:49.702258 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.702170    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9db4c4959-dgbt8" event={"ID":"fa7fa2e8-a4fe-4969-973d-19bad30e8533","Type":"ContainerDied","Data":"d874e45c0c6591c32e53f1e4da601c918f37b8f6d3a04d189f51abbc221ea9a8"}
Oct 30 21:55:49.702436 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.702269    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9db4c4959-dgbt8"
Oct 30 21:55:49.719643 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.719613    5031 scope.go:122] "RemoveContainer" containerID="e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739"
Oct 30 21:55:49.720053 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:55:49.720006    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739\": container with ID starting with e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739 not found: ID does not exist" containerID="e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739"
Oct 30 21:55:49.720135 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.720044    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739"} err="failed to get container status \"e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739\": rpc error: code = NotFound desc = could not find container \"e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739\": container with ID starting with e89f3290f5e4f5668f867519a67ff21dd247dd8773eb998bffa446e16168e739 not found: ID does not exist"
Oct 30 21:55:49.720135 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.720068    5031 scope.go:122] "RemoveContainer" containerID="14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631"
Oct 30 21:55:49.732340 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.732324    5031 scope.go:122] "RemoveContainer" containerID="14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631"
Oct 30 21:55:49.733817 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:55:49.732913    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631\": container with ID starting with 14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631 not found: ID does not exist" containerID="14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631"
Oct 30 21:55:49.733817 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.732962    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631"} err="failed to get container status \"14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631\": rpc error: code = NotFound desc = could not find container \"14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631\": container with ID starting with 14b4b38d70a67141004eee49e95d822685dfbe7f44a91e45fc60672b104c4631 not found: ID does not exist"
Oct 30 21:55:49.742802 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.742782    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:49.781899 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.781878    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z"
Oct 30 21:55:49.886619 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.886538    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99bc674-28d0-4503-8cc4-1334e1ed69aa-kube-api-access-w2f5x\" (UniqueName: \"kubernetes.io/projected/a99bc674-28d0-4503-8cc4-1334e1ed69aa-kube-api-access-w2f5x\") pod \"a99bc674-28d0-4503-8cc4-1334e1ed69aa\" (UID: \"a99bc674-28d0-4503-8cc4-1334e1ed69aa\") "
Oct 30 21:55:49.889508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.889475    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99bc674-28d0-4503-8cc4-1334e1ed69aa-kube-api-access-w2f5x" pod "a99bc674-28d0-4503-8cc4-1334e1ed69aa" (UID: "a99bc674-28d0-4503-8cc4-1334e1ed69aa"). InnerVolumeSpecName "kube-api-access-w2f5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:49.987551 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.987509    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2f5x\" (UniqueName: \"kubernetes.io/projected/a99bc674-28d0-4503-8cc4-1334e1ed69aa-kube-api-access-w2f5x\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:49.994848 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:49.994821    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:50.057388 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.057360    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9db4c4959-dgbt8"]
Oct 30 21:55:50.060274 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.060251    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9db4c4959-dgbt8"]
Oct 30 21:55:50.169239 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.168802    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5hl8c"]
Oct 30 21:55:50.171437 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.171418    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5hl8c"]
Oct 30 21:55:50.228232 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.227603    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:50.461505 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.461427    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:50.559586 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:55:50.559549    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:55:50.644931 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.644901    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a18789c-e0f6-4d05-b12b-34226d2c2ec8" path="/var/lib/kubelet/pods/4a18789c-e0f6-4d05-b12b-34226d2c2ec8/volumes"
Oct 30 21:55:50.645266 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.645252    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7fa2e8-a4fe-4969-973d-19bad30e8533" path="/var/lib/kubelet/pods/fa7fa2e8-a4fe-4969-973d-19bad30e8533/volumes"
Oct 30 21:55:50.728720 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.728628    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z" event={"ID":"a99bc674-28d0-4503-8cc4-1334e1ed69aa","Type":"ContainerDied","Data":"2f3e6e3243c13a9f53d292d246015311796aeacb5412d9ff715a92f1713f75bc"}
Oct 30 21:55:50.732364 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.732336    5031 scope.go:122] "RemoveContainer" containerID="9e73ef7be20e1b8cabee06c414d90b70aca3ccfffdb3253602bfcd67dc687db7"
Oct 30 21:55:50.732593 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.729314    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8k2z"
Oct 30 21:55:50.736268 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.735809    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" event={"ID":"de2775d7-88bd-440d-8034-5a7412a22128","Type":"ContainerStarted","Data":"32ca4c8d3158bb8b56144f3a9596e45d45e2560cf1102747d611c4f19876a93d"}
Oct 30 21:55:50.902032 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.902005    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8k2z"]
Oct 30 21:55:50.908245 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:50.907867    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8k2z"]
Oct 30 21:55:51.160716 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:51.160685    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:55:51.160930 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:51.160907    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kkskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:55:51.161017 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:51.160962    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podUID="de2775d7-88bd-440d-8034-5a7412a22128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 21:55:52.160778 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:52.160745    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kkskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:55:52.161155 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:52.160796    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podUID="de2775d7-88bd-440d-8034-5a7412a22128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 21:55:52.644540 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:52.644505    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99bc674-28d0-4503-8cc4-1334e1ed69aa" path="/var/lib/kubelet/pods/a99bc674-28d0-4503-8cc4-1334e1ed69aa/volumes"
Oct 30 21:55:53.160521 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:53.160487    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kkskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:55:53.160675 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:53.160535    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podUID="de2775d7-88bd-440d-8034-5a7412a22128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 21:55:53.849816 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:53.849786    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:54.078370 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:54.078341    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:54.161355 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:54.161267    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kkskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:55:54.161355 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:54.161329    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podUID="de2775d7-88bd-440d-8034-5a7412a22128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 21:55:55.161256 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:55.161225    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kkskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:55:55.161727 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:55.161692    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podUID="de2775d7-88bd-440d-8034-5a7412a22128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 21:55:56.164399 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:56.164365    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:55:56.164880 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:56.164753    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:55:56.165361 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:56.165341    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg"
Oct 30 21:55:58.687347 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:58.687316    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-9fbbk"
Oct 30 21:55:58.690372 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:55:58.690348    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-92hjx"
Oct 30 21:56:20.540006 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:56:20.539962    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:56:40.796175 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:56:40.796141    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:50.539879 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:56:50.539838    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:57:20.540405 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:57:20.540367    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:57:24.760384 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:24.760348    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:24.994033 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:24.993999    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:41.237901 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:41.237867    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:41.490613 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:41.490540    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:42.920829 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.920800    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"]
Oct 30 21:57:42.921280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921236    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a99bc674-28d0-4503-8cc4-1334e1ed69aa" containerName="authorino"
Oct 30 21:57:42.921280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921252    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99bc674-28d0-4503-8cc4-1334e1ed69aa" containerName="authorino"
Oct 30 21:57:42.921280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921265    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fa7fa2e8-a4fe-4969-973d-19bad30e8533" containerName="authorino"
Oct 30 21:57:42.921280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921274    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7fa2e8-a4fe-4969-973d-19bad30e8533" containerName="authorino"
Oct 30 21:57:42.921407 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921288    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4a18789c-e0f6-4d05-b12b-34226d2c2ec8" containerName="authorino"
Oct 30 21:57:42.921407 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921294    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a18789c-e0f6-4d05-b12b-34226d2c2ec8" containerName="authorino"
Oct 30 21:57:42.921407 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921362    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99bc674-28d0-4503-8cc4-1334e1ed69aa" containerName="authorino"
Oct 30 21:57:42.921407 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921368    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7fa2e8-a4fe-4969-973d-19bad30e8533" containerName="authorino"
Oct 30 21:57:42.921407 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.921378    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a18789c-e0f6-4d05-b12b-34226d2c2ec8" containerName="authorino"
Oct 30 21:57:42.924426 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:42.924411    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.003961 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.003927    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8thv\" (UniqueName: \"kubernetes.io/projected/8a6d5c6a-ec67-4875-960c-916306762084-kube-api-access-j8thv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.004121 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.003967    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.004121 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.003992    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.004121 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.004103    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.004240 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.004138    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.004283 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.004199    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6d5c6a-ec67-4875-960c-916306762084-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.028954 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.028929    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hwt6r\""
Oct 30 21:57:43.029149 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.029131    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:57:43.029187 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.029139    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 21:57:43.029187 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.029156    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:57:43.031403 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.031384    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"]
Oct 30 21:57:43.083105 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.083072    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:43.105381 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105352    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8thv\" (UniqueName: \"kubernetes.io/projected/8a6d5c6a-ec67-4875-960c-916306762084-kube-api-access-j8thv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.105558 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105392    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.105558 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105414    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.105558 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105444    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.105558 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105465    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.105558 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105499    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6d5c6a-ec67-4875-960c-916306762084-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.105908 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105879    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.105908 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.105903    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.106228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.106119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.107692 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.107673    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6d5c6a-ec67-4875-960c-916306762084-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.107963 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.107947    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6d5c6a-ec67-4875-960c-916306762084-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.169343 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.169311    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8thv\" (UniqueName: \"kubernetes.io/projected/8a6d5c6a-ec67-4875-960c-916306762084-kube-api-access-j8thv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn\" (UID: \"8a6d5c6a-ec67-4875-960c-916306762084\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.238386 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.238300    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:43.616810 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.616785    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"]
Oct 30 21:57:43.617054 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.617039    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:43.618371 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:57:43.618348    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6d5c6a_ec67_4875_960c_916306762084.slice/crio-781894935c47b9c481c177987b8a80e8ed0eac089b4536d7d430e5ceb1133207 WatchSource:0}: Error finding container 781894935c47b9c481c177987b8a80e8ed0eac089b4536d7d430e5ceb1133207: Status 404 returned error can't find the container with id 781894935c47b9c481c177987b8a80e8ed0eac089b4536d7d430e5ceb1133207
Oct 30 21:57:43.843146 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:43.843109    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:44.319596 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:44.319559    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn" event={"ID":"8a6d5c6a-ec67-4875-960c-916306762084","Type":"ContainerStarted","Data":"781894935c47b9c481c177987b8a80e8ed0eac089b4536d7d430e5ceb1133207"}
Oct 30 21:57:45.326380 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:45.326340    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn" event={"ID":"8a6d5c6a-ec67-4875-960c-916306762084","Type":"ContainerStarted","Data":"56aed891167113ef44b78d4b10a8f5e7e3ce02fbf26958d5c6c3754c737fe525"}
Oct 30 21:57:45.466994 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:45.466882    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn" podStartSLOduration=1.992185536 podStartE2EDuration="3.466863114s" totalImagesPullingTime="1.474677578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:45.465850088 +0000 UTC m=+476.989992851" watchObservedRunningTime="2026-10-30 21:57:45.466863114 +0000 UTC m=+476.991005876"
Oct 30 21:57:46.239170 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.239133    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:46.584329 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.584248    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:57:46.784906 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.784868    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"]
Oct 30 21:57:46.788503 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.788479    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.849102 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.849040    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.849549 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.849476    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.849707 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.849532    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzvm\" (UniqueName: \"kubernetes.io/projected/ed248874-db33-429b-b509-2a75e3e72304-kube-api-access-tpzvm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.849950 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.849900    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.850058 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.849987    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 21:57:46.850228 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.850166    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.850386 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.850343    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed248874-db33-429b-b509-2a75e3e72304-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.901768 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.901735    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"]
Oct 30 21:57:46.951258 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951219    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.951456 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951273    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzvm\" (UniqueName: \"kubernetes.io/projected/ed248874-db33-429b-b509-2a75e3e72304-kube-api-access-tpzvm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.951456 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951331    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.951564 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951495    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.951564 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951539    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed248874-db33-429b-b509-2a75e3e72304-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.951668 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951597    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.951801 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951772    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.952020 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.951990    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.952501 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.952432    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.954275 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.954254    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed248874-db33-429b-b509-2a75e3e72304-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:46.954421 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:46.954398    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed248874-db33-429b-b509-2a75e3e72304-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:47.023027 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:47.022996    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzvm\" (UniqueName: \"kubernetes.io/projected/ed248874-db33-429b-b509-2a75e3e72304-kube-api-access-tpzvm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c4r4m\" (UID: \"ed248874-db33-429b-b509-2a75e3e72304\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:47.030375 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:47.030345    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:47.102471 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:47.102373    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:47.499867 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:47.499839    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"]
Oct 30 21:57:47.501948 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:57:47.501917    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded248874_db33_429b_b509_2a75e3e72304.slice/crio-9aa80dcf0904490feb1e3e3f15bae311bdcabb6e66e6a68b8570b0e7a95aa46a WatchSource:0}: Error finding container 9aa80dcf0904490feb1e3e3f15bae311bdcabb6e66e6a68b8570b0e7a95aa46a: Status 404 returned error can't find the container with id 9aa80dcf0904490feb1e3e3f15bae311bdcabb6e66e6a68b8570b0e7a95aa46a
Oct 30 21:57:48.344647 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:48.344614    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m" event={"ID":"ed248874-db33-429b-b509-2a75e3e72304","Type":"ContainerStarted","Data":"46f04d4c078cd98b37efb427c33043beaed3ba93daae5ad32ddb97e210ae3121"}
Oct 30 21:57:48.345010 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:48.344677    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m" event={"ID":"ed248874-db33-429b-b509-2a75e3e72304","Type":"ContainerStarted","Data":"9aa80dcf0904490feb1e3e3f15bae311bdcabb6e66e6a68b8570b0e7a95aa46a"}
Oct 30 21:57:48.471120 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:48.471065    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m" podStartSLOduration=1.887246755 podStartE2EDuration="2.471051353s" totalImagesPullingTime="583.804598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:48.470733441 +0000 UTC m=+479.994876206" watchObservedRunningTime="2026-10-30 21:57:48.471051353 +0000 UTC m=+479.995194115"
Oct 30 21:57:49.102970 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:49.102933    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:50.033183 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.033153    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"]
Oct 30 21:57:50.036784 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.036768    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.141033 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.141005    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 21:57:50.144285 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.144258    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"]
Oct 30 21:57:50.184866 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.184828    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1368ca02-43fe-4181-b9e4-933df7a9665e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.185034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.184881    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.185034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.184970    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.185034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.185016    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqrd\" (UniqueName: \"kubernetes.io/projected/1368ca02-43fe-4181-b9e4-933df7a9665e-kube-api-access-qsqrd\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.185157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.185037    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.185157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.185106    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.235377 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.235348    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:50.285851 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.285782    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.285851 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.285819    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.286071 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.285854    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqrd\" (UniqueName: \"kubernetes.io/projected/1368ca02-43fe-4181-b9e4-933df7a9665e-kube-api-access-qsqrd\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.286071 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.285886    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.286071 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.285914    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.286071 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.285956    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1368ca02-43fe-4181-b9e4-933df7a9665e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.286303 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.286202    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.286303 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.286291    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.286370 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.286354    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.288299 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.288277    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1368ca02-43fe-4181-b9e4-933df7a9665e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.288548 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.288532    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1368ca02-43fe-4181-b9e4-933df7a9665e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.349693 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.349660    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqrd\" (UniqueName: \"kubernetes.io/projected/1368ca02-43fe-4181-b9e4-933df7a9665e-kube-api-access-qsqrd\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5grw9\" (UID: \"1368ca02-43fe-4181-b9e4-933df7a9665e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:50.540194 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:57:50.540110    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:57:50.649344 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:50.649315    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:51.048835 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:51.048807    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"]
Oct 30 21:57:51.050790 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:57:51.050763    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1368ca02_43fe_4181_b9e4_933df7a9665e.slice/crio-631eb19d93cd1cbf1ebce83f6d832484f8ff7adede98903aea54755df0c7825e WatchSource:0}: Error finding container 631eb19d93cd1cbf1ebce83f6d832484f8ff7adede98903aea54755df0c7825e: Status 404 returned error can't find the container with id 631eb19d93cd1cbf1ebce83f6d832484f8ff7adede98903aea54755df0c7825e
Oct 30 21:57:51.361184 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:51.361155    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9" event={"ID":"1368ca02-43fe-4181-b9e4-933df7a9665e","Type":"ContainerStarted","Data":"631eb19d93cd1cbf1ebce83f6d832484f8ff7adede98903aea54755df0c7825e"}
Oct 30 21:57:52.375969 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:52.375926    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9" event={"ID":"1368ca02-43fe-4181-b9e4-933df7a9665e","Type":"ContainerStarted","Data":"d3e851c3d9a7e161ea1fccd9617929959982bdf6860587c99a719bb66d399ce2"}
Oct 30 21:57:52.497995 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:52.497938    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9" podStartSLOduration=2.931771107 podStartE2EDuration="3.497925182s" totalImagesPullingTime="566.154075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:52.496882959 +0000 UTC m=+484.021025747" watchObservedRunningTime="2026-10-30 21:57:52.497925182 +0000 UTC m=+484.022067944"
Oct 30 21:57:52.650170 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:52.650096    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:57:53.162973 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.162925    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"]
Oct 30 21:57:53.166730 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.166697    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.227003 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.226973    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 21:57:53.240078 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.239296    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:53.254990 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.254241    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:53.267875 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.267840    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"]
Oct 30 21:57:53.312913 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.312868    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.313108 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.312991    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.313192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.313110    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bfc710-4ddf-4e96-a333-ead674e668ed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.313192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.313147    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.313357 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.313335    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.313421 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.313401    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbc9k\" (UniqueName: \"kubernetes.io/projected/52bfc710-4ddf-4e96-a333-ead674e668ed-kube-api-access-zbc9k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.379420 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.379369    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:53.391846 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.391821    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn"
Oct 30 21:57:53.414602 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.414513    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.414602 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.414591    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bfc710-4ddf-4e96-a333-ead674e668ed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.414807 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.414623    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.414807 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.414698    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.414807 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.414732    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbc9k\" (UniqueName: \"kubernetes.io/projected/52bfc710-4ddf-4e96-a333-ead674e668ed-kube-api-access-zbc9k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.414807 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.414778    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.415038 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.414887    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.415242 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.415204    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.415365 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.415318    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.417530 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.417508    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bfc710-4ddf-4e96-a333-ead674e668ed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.418374 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.418348    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bfc710-4ddf-4e96-a333-ead674e668ed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.481547 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.481512    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbc9k\" (UniqueName: \"kubernetes.io/projected/52bfc710-4ddf-4e96-a333-ead674e668ed-kube-api-access-zbc9k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv\" (UID: \"52bfc710-4ddf-4e96-a333-ead674e668ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:53.781605 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:53.781509    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:54.159871 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:54.159848    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"]
Oct 30 21:57:54.161015 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:57:54.160992    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bfc710_4ddf_4e96_a333_ead674e668ed.slice/crio-4042f119d8814feb25cacfd9070ad68dc35d155b512128447ef570c6be17443f WatchSource:0}: Error finding container 4042f119d8814feb25cacfd9070ad68dc35d155b512128447ef570c6be17443f: Status 404 returned error can't find the container with id 4042f119d8814feb25cacfd9070ad68dc35d155b512128447ef570c6be17443f
Oct 30 21:57:54.389188 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:54.389157    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv" event={"ID":"52bfc710-4ddf-4e96-a333-ead674e668ed","Type":"ContainerStarted","Data":"4042f119d8814feb25cacfd9070ad68dc35d155b512128447ef570c6be17443f"}
Oct 30 21:57:55.396191 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:55.396154    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv" event={"ID":"52bfc710-4ddf-4e96-a333-ead674e668ed","Type":"ContainerStarted","Data":"eadb10bdd5ef173181ca7026b2ef27ee2ea1e4889837ddd340c84f0839b76a2d"}
Oct 30 21:57:55.518190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:55.518141    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv" podStartSLOduration=1.924267825 podStartE2EDuration="2.518127067s" totalImagesPullingTime="593.859242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:55.517031795 +0000 UTC m=+487.041174574" watchObservedRunningTime="2026-10-30 21:57:55.518127067 +0000 UTC m=+487.042269831"
Oct 30 21:57:55.782645 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:55.782572    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:57:57.102714 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.102688    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:57.114901 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.114881    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:57.335045 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.335010    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-gdzbd"]
Oct 30 21:57:57.338853 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.338835    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.398406 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.398320    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 21:57:57.408670 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.408641    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-gdzbd"]
Oct 30 21:57:57.419594 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.419572    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m"
Oct 30 21:57:57.455395 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.455371    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.455519 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.455407    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-home\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.455519 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.455427    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.455519 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.455450    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.455519 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.455504    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7xjt\" (UniqueName: \"kubernetes.io/projected/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-kube-api-access-x7xjt\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.455673 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.455660    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.556884 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.556830    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-home\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.556884 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.556884    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.557123 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.556905    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.557123 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.556924    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7xjt\" (UniqueName: \"kubernetes.io/projected/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-kube-api-access-x7xjt\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.557123 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.557031    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.557123 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.557090    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.557440 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.557414    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-home\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.557508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.557438    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.557508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.557460    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.559548 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.559519    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.559656 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.559629    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.583109 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.583089    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:57.625149 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.625124    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7xjt\" (UniqueName: \"kubernetes.io/projected/06f5935f-a491-4b89-a3a5-bec3aa0f8b26-kube-api-access-x7xjt\") pod \"echo-simulator-kserve-64b6f8f96c-gdzbd\" (UID: \"06f5935f-a491-4b89-a3a5-bec3aa0f8b26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:57.651535 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:57.651473    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:57:58.027025 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:58.026995    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-gdzbd"]
Oct 30 21:57:58.028356 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:57:58.028331    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f5935f_a491_4b89_a3a5_bec3aa0f8b26.slice/crio-f91d760c9f562629dc72016aab8b3c0822844b43b947005d2548eccd92aece82 WatchSource:0}: Error finding container f91d760c9f562629dc72016aab8b3c0822844b43b947005d2548eccd92aece82: Status 404 returned error can't find the container with id f91d760c9f562629dc72016aab8b3c0822844b43b947005d2548eccd92aece82
Oct 30 21:57:58.414807 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:57:58.414774    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd" event={"ID":"06f5935f-a491-4b89-a3a5-bec3aa0f8b26","Type":"ContainerStarted","Data":"f91d760c9f562629dc72016aab8b3c0822844b43b947005d2548eccd92aece82"}
Oct 30 21:58:00.292024 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:00.291948    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:58:00.429871 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:00.429837    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd" event={"ID":"06f5935f-a491-4b89-a3a5-bec3aa0f8b26","Type":"ContainerStarted","Data":"c61e9cda17342b866c0ad927b6bda0bf86c085dbea8c9e46604d0e28109294be"}
Oct 30 21:58:00.554410 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:00.554297    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd" podStartSLOduration=1.666012434 podStartE2EDuration="3.554282239s" totalImagesPullingTime="1.888269805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:00.552508054 +0000 UTC m=+492.076650835" watchObservedRunningTime="2026-10-30 21:58:00.554282239 +0000 UTC m=+492.078425034"
Oct 30 21:58:00.650160 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:00.650133    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:58:00.651797 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:00.651764    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:58:00.663449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:00.663429    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:58:01.455496 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:01.455464    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5grw9"
Oct 30 21:58:02.654964 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:02.654929    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:03.781959 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:03.781927    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:58:03.794592 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:03.794565    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:58:04.465568 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:04.465540    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv"
Oct 30 21:58:07.651830 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:07.651798    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:58:07.652775 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:07.652747    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:07.667978 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:07.667950    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:58:08.490578 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:08.490544    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-gdzbd"
Oct 30 21:58:12.777850 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:12.777815    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:58:14.722533 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:14.722469    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:15.901571 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:15.901541    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:20.540237 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:58:20.540179    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:58:24.589019 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:24.588982    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 21:58:32.616918 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.616886    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"]
Oct 30 21:58:32.624354 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.624325    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675087 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675055    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675265 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675095    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9bg\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-kube-api-access-8g9bg\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675265 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675117    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675389 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675279    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae72439-1274-4abd-8182-6f918935d93f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675389 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675323    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675492 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675417    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675492 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675447    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae72439-1274-4abd-8182-6f918935d93f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675492 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675485    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.675636 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.675546    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.721845 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.721818    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"]
Oct 30 21:58:32.776165 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776139    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776324 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776173    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9bg\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-kube-api-access-8g9bg\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776324 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776193    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776324 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776253    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae72439-1274-4abd-8182-6f918935d93f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776324 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776311    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776536 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776351    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776536 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776380    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae72439-1274-4abd-8182-6f918935d93f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776536 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776445    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776695 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776547    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776749 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776717    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776842 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776820    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776907 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776845    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.776907 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776883    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.777160 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.776997    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae72439-1274-4abd-8182-6f918935d93f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.778636 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.778615    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.778808 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.778792    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae72439-1274-4abd-8182-6f918935d93f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.840495 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.840468    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9bg\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-kube-api-access-8g9bg\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.840602 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.840550    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-k2nvr\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:32.937047 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:32.936977    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:33.365483 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.365458    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"]
Oct 30 21:58:33.366295 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:58:33.366268    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae72439_1274_4abd_8182_6f918935d93f.slice/crio-cf001b17f7bfb5dfe504556ec05511627d80963e12908f618e9a01fb7d57a8d0 WatchSource:0}: Error finding container cf001b17f7bfb5dfe504556ec05511627d80963e12908f618e9a01fb7d57a8d0: Status 404 returned error can't find the container with id cf001b17f7bfb5dfe504556ec05511627d80963e12908f618e9a01fb7d57a8d0
Oct 30 21:58:33.368434 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.368404    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:33.368514 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.368468    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:33.368514 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.368499    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:58:33.622530 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.622442    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" event={"ID":"cae72439-1274-4abd-8182-6f918935d93f","Type":"ContainerStarted","Data":"9dfd65acf94c4e4eca24eceed2632bd4c8cd8173e90ac798930886a415145270"}
Oct 30 21:58:33.622530 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.622505    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" event={"ID":"cae72439-1274-4abd-8182-6f918935d93f","Type":"ContainerStarted","Data":"cf001b17f7bfb5dfe504556ec05511627d80963e12908f618e9a01fb7d57a8d0"}
Oct 30 21:58:33.748187 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.748130    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podStartSLOduration=1.748116874 podStartE2EDuration="1.748116874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:33.746525166 +0000 UTC m=+525.270667941" watchObservedRunningTime="2026-10-30 21:58:33.748116874 +0000 UTC m=+525.272259636"
Oct 30 21:58:33.937618 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:33.937519    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:34.938933 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:34.938898    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:58:34.939429 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:34.938957    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:58:35.937596 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:35.937559    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:58:35.937796 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:35.937612    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:58:36.938277 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:36.938247    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:58:36.938685 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:36.938309    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:58:37.498531 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.498499    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bbc6dbb5d-4jll5"]
Oct 30 21:58:37.503294 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.503273    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.559418 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.559388    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bbc6dbb5d-4jll5"]
Oct 30 21:58:37.612234 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.608585    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-86hk6\""
Oct 30 21:58:37.612234 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.608680    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 21:58:37.637023 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.636992    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faf7f082-2144-4538-959e-5f30b01c7aa1-tls-cert\") pod \"authorino-5bbc6dbb5d-4jll5\" (UID: \"faf7f082-2144-4538-959e-5f30b01c7aa1\") " pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.637301 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.637167    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9j6\" (UniqueName: \"kubernetes.io/projected/faf7f082-2144-4538-959e-5f30b01c7aa1-kube-api-access-hf9j6\") pod \"authorino-5bbc6dbb5d-4jll5\" (UID: \"faf7f082-2144-4538-959e-5f30b01c7aa1\") " pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.671278 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.671246    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"]
Oct 30 21:58:37.675636 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.675613    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:37.730237 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.728700    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:37.732551 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.732523    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 21:58:37.736420 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.736398    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-zwb2w\""
Oct 30 21:58:37.737192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.737142    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 21:58:37.739295 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.737797    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faf7f082-2144-4538-959e-5f30b01c7aa1-tls-cert\") pod \"authorino-5bbc6dbb5d-4jll5\" (UID: \"faf7f082-2144-4538-959e-5f30b01c7aa1\") " pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.739514 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.739489    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdx5\" (UniqueName: \"kubernetes.io/projected/76fe2101-7e44-4365-8d9a-bae44a219330-kube-api-access-7xdx5\") pod \"ai-gateway-controller-66dd76bf94-q5r6r\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:37.739916 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.739545    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fe2101-7e44-4365-8d9a-bae44a219330-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-q5r6r\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:37.739916 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.739699    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9j6\" (UniqueName: \"kubernetes.io/projected/faf7f082-2144-4538-959e-5f30b01c7aa1-kube-api-access-hf9j6\") pod \"authorino-5bbc6dbb5d-4jll5\" (UID: \"faf7f082-2144-4538-959e-5f30b01c7aa1\") " pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.740765 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.740599    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"]
Oct 30 21:58:37.743676 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.743657    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faf7f082-2144-4538-959e-5f30b01c7aa1-tls-cert\") pod \"authorino-5bbc6dbb5d-4jll5\" (UID: \"faf7f082-2144-4538-959e-5f30b01c7aa1\") " pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.832335 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.832258    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9j6\" (UniqueName: \"kubernetes.io/projected/faf7f082-2144-4538-959e-5f30b01c7aa1-kube-api-access-hf9j6\") pod \"authorino-5bbc6dbb5d-4jll5\" (UID: \"faf7f082-2144-4538-959e-5f30b01c7aa1\") " pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.841093 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.841067    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdx5\" (UniqueName: \"kubernetes.io/projected/76fe2101-7e44-4365-8d9a-bae44a219330-kube-api-access-7xdx5\") pod \"ai-gateway-controller-66dd76bf94-q5r6r\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:37.841260 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.841103    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fe2101-7e44-4365-8d9a-bae44a219330-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-q5r6r\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:37.843800 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.843769    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fe2101-7e44-4365-8d9a-bae44a219330-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-q5r6r\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:37.851634 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.851611    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5"
Oct 30 21:58:37.906020 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.905994    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdx5\" (UniqueName: \"kubernetes.io/projected/76fe2101-7e44-4365-8d9a-bae44a219330-kube-api-access-7xdx5\") pod \"ai-gateway-controller-66dd76bf94-q5r6r\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:37.939388 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.938891    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:58:37.939388 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.938980    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:58:37.981493 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.981462    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:37.988396 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:37.988371    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:38.236794 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:38.236762    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bbc6dbb5d-4jll5"]
Oct 30 21:58:38.241728 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:58:38.241702    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaf7f082_2144_4538_959e_5f30b01c7aa1.slice/crio-48ba1c72476b460e89c4e7f7e16ce98f1d93298016f24903d4c5150bd9064b8f WatchSource:0}: Error finding container 48ba1c72476b460e89c4e7f7e16ce98f1d93298016f24903d4c5150bd9064b8f: Status 404 returned error can't find the container with id 48ba1c72476b460e89c4e7f7e16ce98f1d93298016f24903d4c5150bd9064b8f
Oct 30 21:58:38.393613 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:38.393588    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"]
Oct 30 21:58:38.394680 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:58:38.394646    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fe2101_7e44_4365_8d9a_bae44a219330.slice/crio-766311cb2a6e5822a5588425cc5af767cc62ed265be8668c523daf319bfbcc2c WatchSource:0}: Error finding container 766311cb2a6e5822a5588425cc5af767cc62ed265be8668c523daf319bfbcc2c: Status 404 returned error can't find the container with id 766311cb2a6e5822a5588425cc5af767cc62ed265be8668c523daf319bfbcc2c
Oct 30 21:58:38.657063 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:38.657024    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r" event={"ID":"76fe2101-7e44-4365-8d9a-bae44a219330","Type":"ContainerStarted","Data":"766311cb2a6e5822a5588425cc5af767cc62ed265be8668c523daf319bfbcc2c"}
Oct 30 21:58:38.658255 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:38.658229    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5" event={"ID":"faf7f082-2144-4538-959e-5f30b01c7aa1","Type":"ContainerStarted","Data":"48ba1c72476b460e89c4e7f7e16ce98f1d93298016f24903d4c5150bd9064b8f"}
Oct 30 21:58:38.937563 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:38.937481    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:58:38.937563 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:38.937532    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:58:39.620036 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:39.620006    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:39.938336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:39.938268    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:58:39.938449 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:39.938331    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:58:40.681163 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:40.681120    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5" event={"ID":"faf7f082-2144-4538-959e-5f30b01c7aa1","Type":"ContainerStarted","Data":"fdb1701de04253b553c69cada51d1e9b04e7efc14476d0a94f18c9e1e6d59f57"}
Oct 30 21:58:40.804832 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:40.804772    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5bbc6dbb5d-4jll5" podStartSLOduration=2.211385951 podStartE2EDuration="3.804757863s" totalImagesPullingTime="1.593371912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:40.802505781 +0000 UTC m=+532.326648555" watchObservedRunningTime="2026-10-30 21:58:40.804757863 +0000 UTC m=+532.328900654"
Oct 30 21:58:41.514232 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:41.513580    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:41.871504 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:41.871474    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:41.872655 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:41.872629    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:41.873375 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:41.873348    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 21:58:42.695897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:42.695858    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r" event={"ID":"76fe2101-7e44-4365-8d9a-bae44a219330","Type":"ContainerStarted","Data":"e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b"}
Oct 30 21:58:42.817280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:42.817187    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r" podStartSLOduration=2.207766658 podStartE2EDuration="5.817166997s" totalImagesPullingTime="3.609400339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:42.816408548 +0000 UTC m=+534.340551337" watchObservedRunningTime="2026-10-30 21:58:42.817166997 +0000 UTC m=+534.341309761"
Oct 30 21:58:43.702428 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:43.702389    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:44.869288 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:44.869190    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:45.092015 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.091951    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:45.223403 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.223329    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"]
Oct 30 21:58:45.226802 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.226786    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.283356 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.283304    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 21:58:45.283570 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.283523    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gh5hk\""
Oct 30 21:58:45.326850 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.326814    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwhr\" (UniqueName: \"kubernetes.io/projected/2007da67-21d3-4b4a-aa73-69ad80f82250-kube-api-access-dpwhr\") pod \"payload-pre-processing-7b98bb678c-f82bg\" (UID: \"2007da67-21d3-4b4a-aa73-69ad80f82250\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.327113 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.326966    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2007da67-21d3-4b4a-aa73-69ad80f82250-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f82bg\" (UID: \"2007da67-21d3-4b4a-aa73-69ad80f82250\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.338048 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.338002    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"]
Oct 30 21:58:45.354752 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.354721    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wztzx"]
Oct 30 21:58:45.358534 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.358511    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:45.417246 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.417176    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wztzx"]
Oct 30 21:58:45.432003 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.430854    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwhr\" (UniqueName: \"kubernetes.io/projected/2007da67-21d3-4b4a-aa73-69ad80f82250-kube-api-access-dpwhr\") pod \"payload-pre-processing-7b98bb678c-f82bg\" (UID: \"2007da67-21d3-4b4a-aa73-69ad80f82250\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.432003 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.431061    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2007da67-21d3-4b4a-aa73-69ad80f82250-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f82bg\" (UID: \"2007da67-21d3-4b4a-aa73-69ad80f82250\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.432199 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.432110    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2007da67-21d3-4b4a-aa73-69ad80f82250-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f82bg\" (UID: \"2007da67-21d3-4b4a-aa73-69ad80f82250\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.494321 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.494248    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwhr\" (UniqueName: \"kubernetes.io/projected/2007da67-21d3-4b4a-aa73-69ad80f82250-kube-api-access-dpwhr\") pod \"payload-pre-processing-7b98bb678c-f82bg\" (UID: \"2007da67-21d3-4b4a-aa73-69ad80f82250\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.532306 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.532265    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f526c67-8dfe-4b2e-9c8f-42aa57efebec-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wztzx\" (UID: \"3f526c67-8dfe-4b2e-9c8f-42aa57efebec\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:45.532485 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.532403    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8dh\" (UniqueName: \"kubernetes.io/projected/3f526c67-8dfe-4b2e-9c8f-42aa57efebec-kube-api-access-ks8dh\") pod \"payload-processing-55fbbbf9dc-wztzx\" (UID: \"3f526c67-8dfe-4b2e-9c8f-42aa57efebec\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:45.539165 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.539142    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:45.633343 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.633101    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f526c67-8dfe-4b2e-9c8f-42aa57efebec-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wztzx\" (UID: \"3f526c67-8dfe-4b2e-9c8f-42aa57efebec\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:45.633343 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.633155    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8dh\" (UniqueName: \"kubernetes.io/projected/3f526c67-8dfe-4b2e-9c8f-42aa57efebec-kube-api-access-ks8dh\") pod \"payload-processing-55fbbbf9dc-wztzx\" (UID: \"3f526c67-8dfe-4b2e-9c8f-42aa57efebec\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:45.633949 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.633925    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f526c67-8dfe-4b2e-9c8f-42aa57efebec-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wztzx\" (UID: \"3f526c67-8dfe-4b2e-9c8f-42aa57efebec\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:45.698120 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.698083    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8dh\" (UniqueName: \"kubernetes.io/projected/3f526c67-8dfe-4b2e-9c8f-42aa57efebec-kube-api-access-ks8dh\") pod \"payload-processing-55fbbbf9dc-wztzx\" (UID: \"3f526c67-8dfe-4b2e-9c8f-42aa57efebec\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:45.928783 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.928754    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"]
Oct 30 21:58:45.930380 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:58:45.930349    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2007da67_21d3_4b4a_aa73_69ad80f82250.slice/crio-442662aa16965549c34c6184a3b6bb353816055f6204ff6b61b79a130cac08c8 WatchSource:0}: Error finding container 442662aa16965549c34c6184a3b6bb353816055f6204ff6b61b79a130cac08c8: Status 404 returned error can't find the container with id 442662aa16965549c34c6184a3b6bb353816055f6204ff6b61b79a130cac08c8
Oct 30 21:58:45.972495 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:45.972460    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:46.361985 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:58:46.361939    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f526c67_8dfe_4b2e_9c8f_42aa57efebec.slice/crio-b686aa958aecfd1730aef64559eb99e16dc1df203cc1345f712f5fc896abfe69 WatchSource:0}: Error finding container b686aa958aecfd1730aef64559eb99e16dc1df203cc1345f712f5fc896abfe69: Status 404 returned error can't find the container with id b686aa958aecfd1730aef64559eb99e16dc1df203cc1345f712f5fc896abfe69
Oct 30 21:58:46.362157 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:46.362129    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wztzx"]
Oct 30 21:58:46.724114 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:46.724018    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx" event={"ID":"3f526c67-8dfe-4b2e-9c8f-42aa57efebec","Type":"ContainerStarted","Data":"b686aa958aecfd1730aef64559eb99e16dc1df203cc1345f712f5fc896abfe69"}
Oct 30 21:58:46.725522 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:46.725494    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg" event={"ID":"2007da67-21d3-4b4a-aa73-69ad80f82250","Type":"ContainerStarted","Data":"442662aa16965549c34c6184a3b6bb353816055f6204ff6b61b79a130cac08c8"}
Oct 30 21:58:49.751019 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:49.750982    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx" event={"ID":"3f526c67-8dfe-4b2e-9c8f-42aa57efebec","Type":"ContainerStarted","Data":"fb0df2768daecc7eccacfc9720b7c92e02ab3311bc10783e637126034c3afc81"}
Oct 30 21:58:49.751534 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:49.751058    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:58:49.752561 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:49.752537    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg" event={"ID":"2007da67-21d3-4b4a-aa73-69ad80f82250","Type":"ContainerStarted","Data":"755ac25d1cb800223d94c07eb5157783a280ec8fdfcff8bbf52ede26a88276d7"}
Oct 30 21:58:49.752678 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:49.752584    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:58:49.909532 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:49.909480    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx" podStartSLOduration=2.085598777 podStartE2EDuration="4.909468127s" totalImagesPullingTime="2.82386935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:49.877591365 +0000 UTC m=+541.401734128" watchObservedRunningTime="2026-10-30 21:58:49.909468127 +0000 UTC m=+541.433610886"
Oct 30 21:58:50.540725 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:58:50.540684    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:58:54.569075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:54.569037    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:54.710469 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:54.710441    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:58:54.832281 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:54.832152    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg" podStartSLOduration=6.665840046 podStartE2EDuration="9.83213962s" totalImagesPullingTime="3.166299574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:49.999437724 +0000 UTC m=+541.523580499" watchObservedRunningTime="2026-10-30 21:58:54.83213962 +0000 UTC m=+546.356282377"
Oct 30 21:58:55.083241 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.083152    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"]
Oct 30 21:58:55.087153 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.087136    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.146957 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.146921    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"]
Oct 30 21:58:55.150856 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.150832    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtnt\" (UniqueName: \"kubernetes.io/projected/128bd426-3a53-4314-ad3f-a76d97b84950-kube-api-access-7qtnt\") pod \"ai-gateway-controller-65d9c54ff9-bncj2\" (UID: \"128bd426-3a53-4314-ad3f-a76d97b84950\") " pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.150985 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.150923    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128bd426-3a53-4314-ad3f-a76d97b84950-metrics-certs\") pod \"ai-gateway-controller-65d9c54ff9-bncj2\" (UID: \"128bd426-3a53-4314-ad3f-a76d97b84950\") " pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.251776 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.251742    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtnt\" (UniqueName: \"kubernetes.io/projected/128bd426-3a53-4314-ad3f-a76d97b84950-kube-api-access-7qtnt\") pod \"ai-gateway-controller-65d9c54ff9-bncj2\" (UID: \"128bd426-3a53-4314-ad3f-a76d97b84950\") " pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.251970 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.251827    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128bd426-3a53-4314-ad3f-a76d97b84950-metrics-certs\") pod \"ai-gateway-controller-65d9c54ff9-bncj2\" (UID: \"128bd426-3a53-4314-ad3f-a76d97b84950\") " pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.254310 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.254284    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128bd426-3a53-4314-ad3f-a76d97b84950-metrics-certs\") pod \"ai-gateway-controller-65d9c54ff9-bncj2\" (UID: \"128bd426-3a53-4314-ad3f-a76d97b84950\") " pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.316725 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.316693    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtnt\" (UniqueName: \"kubernetes.io/projected/128bd426-3a53-4314-ad3f-a76d97b84950-kube-api-access-7qtnt\") pod \"ai-gateway-controller-65d9c54ff9-bncj2\" (UID: \"128bd426-3a53-4314-ad3f-a76d97b84950\") " pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.400075 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.400047    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:55.777072 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.777038    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"]
Oct 30 21:58:55.778221 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:58:55.778175    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128bd426_3a53_4314_ad3f_a76d97b84950.slice/crio-27441bbb18225d8a890ab23fc91250bb030d1ec36427d675e4954507ea839f55 WatchSource:0}: Error finding container 27441bbb18225d8a890ab23fc91250bb030d1ec36427d675e4954507ea839f55: Status 404 returned error can't find the container with id 27441bbb18225d8a890ab23fc91250bb030d1ec36427d675e4954507ea839f55
Oct 30 21:58:55.787437 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:55.787410    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2" event={"ID":"128bd426-3a53-4314-ad3f-a76d97b84950","Type":"ContainerStarted","Data":"27441bbb18225d8a890ab23fc91250bb030d1ec36427d675e4954507ea839f55"}
Oct 30 21:58:56.794742 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:56.794702    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2" event={"ID":"128bd426-3a53-4314-ad3f-a76d97b84950","Type":"ContainerStarted","Data":"c3a86db06a6b95a188f59de2ba9c29cd5f0de70ed120bfbd7fc2dcd289a44a8a"}
Oct 30 21:58:56.795166 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:56.794777    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:58:56.916873 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:58:56.916821    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2" podStartSLOduration=0.977891576 podStartE2EDuration="1.916805589s" totalImagesPullingTime="938.914013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:56.916398594 +0000 UTC m=+548.440541354" watchObservedRunningTime="2026-10-30 21:58:56.916805589 +0000 UTC m=+548.440948354"
Oct 30 21:59:00.761768 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:00.761732    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-wztzx"
Oct 30 21:59:00.762190 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:00.761805    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f82bg"
Oct 30 21:59:07.806437 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:07.806404    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-65d9c54ff9-bncj2"
Oct 30 21:59:07.979640 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:07.979590    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"]
Oct 30 21:59:07.979903 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:07.979872    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r" podUID="76fe2101-7e44-4365-8d9a-bae44a219330" containerName="manager" containerID="cri-o://e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b" gracePeriod=30
Oct 30 21:59:08.235379 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.235355    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:59:08.283492 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.283455    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fe2101-7e44-4365-8d9a-bae44a219330-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fe2101-7e44-4365-8d9a-bae44a219330-metrics-certs\") pod \"76fe2101-7e44-4365-8d9a-bae44a219330\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") "
Oct 30 21:59:08.283646 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.283524    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fe2101-7e44-4365-8d9a-bae44a219330-kube-api-access-7xdx5\" (UniqueName: \"kubernetes.io/projected/76fe2101-7e44-4365-8d9a-bae44a219330-kube-api-access-7xdx5\") pod \"76fe2101-7e44-4365-8d9a-bae44a219330\" (UID: \"76fe2101-7e44-4365-8d9a-bae44a219330\") "
Oct 30 21:59:08.285815 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.285788    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fe2101-7e44-4365-8d9a-bae44a219330-metrics-certs" pod "76fe2101-7e44-4365-8d9a-bae44a219330" (UID: "76fe2101-7e44-4365-8d9a-bae44a219330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:08.285918 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.285842    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fe2101-7e44-4365-8d9a-bae44a219330-kube-api-access-7xdx5" pod "76fe2101-7e44-4365-8d9a-bae44a219330" (UID: "76fe2101-7e44-4365-8d9a-bae44a219330"). InnerVolumeSpecName "kube-api-access-7xdx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:08.384584 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.384501    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fe2101-7e44-4365-8d9a-bae44a219330-metrics-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:08.384584 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.384529    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xdx5\" (UniqueName: \"kubernetes.io/projected/76fe2101-7e44-4365-8d9a-bae44a219330-kube-api-access-7xdx5\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:08.866521 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.866486    5031 generic.go:356] "Generic (PLEG): container finished" podID="76fe2101-7e44-4365-8d9a-bae44a219330" containerID="e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b" exitCode=0
Oct 30 21:59:08.866952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.866536    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"
Oct 30 21:59:08.866952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.866575    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r" event={"ID":"76fe2101-7e44-4365-8d9a-bae44a219330","Type":"ContainerDied","Data":"e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b"}
Oct 30 21:59:08.866952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.866631    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-q5r6r" event={"ID":"76fe2101-7e44-4365-8d9a-bae44a219330","Type":"ContainerDied","Data":"766311cb2a6e5822a5588425cc5af767cc62ed265be8668c523daf319bfbcc2c"}
Oct 30 21:59:08.866952 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.866651    5031 scope.go:122] "RemoveContainer" containerID="e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b"
Oct 30 21:59:08.876047 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.876021    5031 scope.go:122] "RemoveContainer" containerID="e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b"
Oct 30 21:59:08.876397 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:59:08.876375    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b\": container with ID starting with e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b not found: ID does not exist" containerID="e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b"
Oct 30 21:59:08.876480 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:08.876406    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b"} err="failed to get container status \"e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b\": rpc error: code = NotFound desc = could not find container \"e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b\": container with ID starting with e017bdec7d77b5d346ef1f387d1c07a552181ba6b8de7f6f32b4cb81ab2e8d8b not found: ID does not exist"
Oct 30 21:59:09.037298 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:09.037192    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"]
Oct 30 21:59:09.087293 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:09.087263    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-q5r6r"]
Oct 30 21:59:09.750445 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:09.750403    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:10.318056 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:10.318006    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:10.645446 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:10.645412    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fe2101-7e44-4365-8d9a-bae44a219330" path="/var/lib/kubelet/pods/76fe2101-7e44-4365-8d9a-bae44a219330/volumes"
Oct 30 21:59:12.003359 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:12.003307    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:12.372582 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:12.372552    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:13.898832 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:13.898788    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:14.146539 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:14.146503    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:20.540347 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:59:20.540301    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:59:27.138663 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:27.138622    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:41.396536 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.396444    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:41.629297 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.629269    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:41.855795 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.855761    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:41.874255 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.874198    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"]
Oct 30 21:59:41.874844 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.874828    5031 cpu_manager.go:414] "containerMap: removing container" podUID="76fe2101-7e44-4365-8d9a-bae44a219330" containerName="manager"
Oct 30 21:59:41.874899 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.874846    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fe2101-7e44-4365-8d9a-bae44a219330" containerName="manager"
Oct 30 21:59:41.874944 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.874936    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fe2101-7e44-4365-8d9a-bae44a219330" containerName="manager"
Oct 30 21:59:41.878446 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.878428    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.916018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.915981    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.916480 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.916378    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.916710 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.916693    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.917004 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.916983    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7hpn\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-kube-api-access-k7hpn\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.917409 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.917381    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.917656 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.917429    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.917896 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.917700    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.918194 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.917959    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.918294 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.918257    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:41.934327 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.934294    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6a0a09-openshift-default-dockercfg-h7ds4\""
Oct 30 21:59:41.986412 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:41.986375    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"]
Oct 30 21:59:42.020726 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.020692    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.020726 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.020729    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.020940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.020759    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.020940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.020790    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.020940 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.020814    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021096 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.020963    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021096 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021025    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021096 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021083    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021260 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021108    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hpn\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-kube-api-access-k7hpn\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021260 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021203    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021360 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021252    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021414 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021357    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021473 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021418    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.021664 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.021639    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.023450 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.023426    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.023635 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.023616    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.084719 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.084685    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.084866 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.084828    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hpn\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-kube-api-access-k7hpn\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.086956 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.086934    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:42.191147 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.191060    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:42.318918 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.318884    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:42.563078 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.562999    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"]
Oct 30 21:59:42.567897 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.567878    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.603474 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.603381    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:42.623828 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.623796    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3783c2-openshift-default-dockercfg-4c7q9\""
Oct 30 21:59:42.631419 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631271    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631419 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631327    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631419 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631396    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rdz\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-kube-api-access-m5rdz\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631442    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631500    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631570    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631596    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631624    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631686 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631681    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.631901 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.631778    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"]
Oct 30 21:59:42.636064 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.636043    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"]
Oct 30 21:59:42.639077 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.638318    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:42.639077 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.638404    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:42.639077 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.638448    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:42.733254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.733078    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rdz\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-kube-api-access-m5rdz\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.733720 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.733585    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.733720 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.733622    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.733720 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.733690    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.734336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.733982    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.734336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.734021    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.734336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.734097    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.734336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.734138    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.734336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.734184    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.739508 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.734916    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.748280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.740557    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.748280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.742483    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.748280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.743232    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.748280 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.745337    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.749872 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.749849    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.750774 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.750739    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.805099 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.805064    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rdz\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-kube-api-access-m5rdz\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.806488 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.806466    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:42.830809 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.830731    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:42.882153 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:42.881910    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:43.065144 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.065112    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" event={"ID":"ac3999f7-0b9d-4cb8-8023-e1703c40c68b","Type":"ContainerStarted","Data":"5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f"}
Oct 30 21:59:43.065355 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.065171    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" event={"ID":"ac3999f7-0b9d-4cb8-8023-e1703c40c68b","Type":"ContainerStarted","Data":"e17c22443f7f47291b9111d4423fa27088dd389064bee09d6c98fb7e0c522d1c"}
Oct 30 21:59:43.101094 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.101067    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:43.190654 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.190606    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" podStartSLOduration=2.190591275 podStartE2EDuration="2.190591275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:43.18981925 +0000 UTC m=+594.713962022" watchObservedRunningTime="2026-10-30 21:59:43.190591275 +0000 UTC m=+594.714734092"
Oct 30 21:59:43.191925 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.191896    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:43.197450 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.197426    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:43.237534 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.237503    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"]
Oct 30 21:59:43.248747 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.248718    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.304045 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.303993    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bce513-openshift-default-dockercfg-jq86d\""
Oct 30 21:59:43.307116 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.307091    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"]
Oct 30 21:59:43.330585 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.330551    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:43.341544 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341468    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341544 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341498    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341544 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341537    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341728 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341554    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341728 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341571    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341728 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341617    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341728 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341654    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341728 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341679    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdhc\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-kube-api-access-9rdhc\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.341728 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.341698    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.343163 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.343141    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"]
Oct 30 21:59:43.344487 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:43.344462    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646d59da_3bfd_4e6a_a6b1_5e387db1bb44.slice/crio-cd1827c2db5f45c8012cebacf587eb45d44a78f8a7b717fe51d9c746321b28fb WatchSource:0}: Error finding container cd1827c2db5f45c8012cebacf587eb45d44a78f8a7b717fe51d9c746321b28fb: Status 404 returned error can't find the container with id cd1827c2db5f45c8012cebacf587eb45d44a78f8a7b717fe51d9c746321b28fb
Oct 30 21:59:43.346932 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.346902    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:43.347028 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.346978    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:43.347028 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.347020    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.442661    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.442710    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.442816    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.442876    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.442902    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.442963    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.443016    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.443046    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdhc\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-kube-api-access-9rdhc\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.443075    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.443434    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.444018 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.443768    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.445510 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.444319    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.445510 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.444416    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.445510 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.444875    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.446077 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.446055    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.447457 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.447281    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.506993 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.506960    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.507174 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.507150    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdhc\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-kube-api-access-9rdhc\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.552709 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.552672    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:43.562652 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.562626    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:43.770673 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.770640    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:43.995140 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:43.995059    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:44.007938 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.007911    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"]
Oct 30 21:59:44.009167 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:44.009146    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6e59d2_78a1_42a9_af56_e4c01eb00496.slice/crio-124221ecbcd14f547a4074e961c109d4ec03902cba9dc527230f79b45d1add5a WatchSource:0}: Error finding container 124221ecbcd14f547a4074e961c109d4ec03902cba9dc527230f79b45d1add5a: Status 404 returned error can't find the container with id 124221ecbcd14f547a4074e961c109d4ec03902cba9dc527230f79b45d1add5a
Oct 30 21:59:44.011505 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.011473    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:44.011573 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.011552    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:44.011608 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.011582    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:59:44.073178 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.073136    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" event={"ID":"646d59da-3bfd-4e6a-a6b1-5e387db1bb44","Type":"ContainerStarted","Data":"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"}
Oct 30 21:59:44.073372 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.073229    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" event={"ID":"646d59da-3bfd-4e6a-a6b1-5e387db1bb44","Type":"ContainerStarted","Data":"cd1827c2db5f45c8012cebacf587eb45d44a78f8a7b717fe51d9c746321b28fb"}
Oct 30 21:59:44.075012 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.074980    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" event={"ID":"1b6e59d2-78a1-42a9-af56-e4c01eb00496","Type":"ContainerStarted","Data":"124221ecbcd14f547a4074e961c109d4ec03902cba9dc527230f79b45d1add5a"}
Oct 30 21:59:44.075258 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.075243    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:44.076483 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.076464    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 21:59:44.217996 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.217964    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:44.331581 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.331477    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" podStartSLOduration=2.331459743 podStartE2EDuration="2.331459743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:44.204759928 +0000 UTC m=+595.728902692" watchObservedRunningTime="2026-10-30 21:59:44.331459743 +0000 UTC m=+595.855602507"
Oct 30 21:59:44.442589 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.442557    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:44.882934 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.882902    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:44.888310 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:44.888286    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:45.085771 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:45.085730    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" event={"ID":"1b6e59d2-78a1-42a9-af56-e4c01eb00496","Type":"ContainerStarted","Data":"4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17"}
Oct 30 21:59:45.086332 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:45.086294    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:45.087071 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:45.087046    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 21:59:45.210437 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:45.210326    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" podStartSLOduration=2.210308268 podStartE2EDuration="2.210308268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:45.209927845 +0000 UTC m=+596.734070633" watchObservedRunningTime="2026-10-30 21:59:45.210308268 +0000 UTC m=+596.734451031"
Oct 30 21:59:45.563528 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:45.563445    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:45.569175 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:45.569150    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:46.092025 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:46.091987    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:46.092813 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:46.092795    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 21:59:46.349548 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:46.349461    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:46.581966 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:46.581925    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:47.431045 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:47.431014    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:47.662036 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:47.662006    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:48.020519 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:48.020483    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:48.286174 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:48.286097    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:48.507982 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:48.507940    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:48.709032 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:48.709008    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:59:48.710117 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:48.710094    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 21:59:48.734047 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:48.733988    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:48.983285 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:48.983190    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:50.541062 ip-10-0-141-147 kubenswrapper[5031]: E1030 21:59:50.541032    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 21:59:50.756055 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:50.756013    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:51.033928 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.033888    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:51.293134 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.293032    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:51.532651 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.532619    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"]
Oct 30 21:59:51.537862 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.537838    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.632992 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.632960    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129685bc-38ce-4438-974d-bb2ea0c1a743-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.633422 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.633013    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftlf\" (UniqueName: \"kubernetes.io/projected/129685bc-38ce-4438-974d-bb2ea0c1a743-kube-api-access-cftlf\") pod \"payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.637084 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.637046    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:51.642529 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.642503    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6a0a09-dockercfg-qmrfj\""
Oct 30 21:59:51.642529 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.642526    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6a0a09\""
Oct 30 21:59:51.645095 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.645075    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"]
Oct 30 21:59:51.682233 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.681894    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"]
Oct 30 21:59:51.686756 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.686702    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:51.732951 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.732914    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"]
Oct 30 21:59:51.735034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.734669    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1820885-4e6b-40aa-9c73-f64219b771dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:51.735034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.734737    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmb4\" (UniqueName: \"kubernetes.io/projected/d1820885-4e6b-40aa-9c73-f64219b771dd-kube-api-access-4xmb4\") pod \"payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:51.735034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.734861    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129685bc-38ce-4438-974d-bb2ea0c1a743-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.735034 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.734937    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftlf\" (UniqueName: \"kubernetes.io/projected/129685bc-38ce-4438-974d-bb2ea0c1a743-kube-api-access-cftlf\") pod \"payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.735887 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.735866    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129685bc-38ce-4438-974d-bb2ea0c1a743-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.800817 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.800743    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftlf\" (UniqueName: \"kubernetes.io/projected/129685bc-38ce-4438-974d-bb2ea0c1a743-kube-api-access-cftlf\") pod \"payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.836103 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.836071    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmb4\" (UniqueName: \"kubernetes.io/projected/d1820885-4e6b-40aa-9c73-f64219b771dd-kube-api-access-4xmb4\") pod \"payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:51.836287 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.836256    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1820885-4e6b-40aa-9c73-f64219b771dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:51.836944 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.836926    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1820885-4e6b-40aa-9c73-f64219b771dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:51.851526 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.851494    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:51.890134 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.890104    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:51.899729 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:51.899703    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmb4\" (UniqueName: \"kubernetes.io/projected/d1820885-4e6b-40aa-9c73-f64219b771dd-kube-api-access-4xmb4\") pod \"payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:52.006001 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:52.005970    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:52.234794 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:52.234768    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"]
Oct 30 21:59:52.235673 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:52.235645    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129685bc_38ce_4438_974d_bb2ea0c1a743.slice/crio-881f1b81be8e8a5e0de9eafdae06f8d75d143c147a65f6ba0e889b46264ad30f WatchSource:0}: Error finding container 881f1b81be8e8a5e0de9eafdae06f8d75d143c147a65f6ba0e889b46264ad30f: Status 404 returned error can't find the container with id 881f1b81be8e8a5e0de9eafdae06f8d75d143c147a65f6ba0e889b46264ad30f
Oct 30 21:59:52.385992 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:52.385951    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1820885_4e6b_40aa_9c73_f64219b771dd.slice/crio-aed17c76a93975d33c66e20d06ed26b38e104b6ad76f1d320a65ced752ea188a WatchSource:0}: Error finding container aed17c76a93975d33c66e20d06ed26b38e104b6ad76f1d320a65ced752ea188a: Status 404 returned error can't find the container with id aed17c76a93975d33c66e20d06ed26b38e104b6ad76f1d320a65ced752ea188a
Oct 30 21:59:52.386643 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:52.386615    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"]
Oct 30 21:59:53.138721 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.138689    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" event={"ID":"129685bc-38ce-4438-974d-bb2ea0c1a743","Type":"ContainerStarted","Data":"aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286"}
Oct 30 21:59:53.139203 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.138746    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" event={"ID":"129685bc-38ce-4438-974d-bb2ea0c1a743","Type":"ContainerStarted","Data":"881f1b81be8e8a5e0de9eafdae06f8d75d143c147a65f6ba0e889b46264ad30f"}
Oct 30 21:59:53.139203 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.138849    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 21:59:53.140580 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.140553    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" event={"ID":"d1820885-4e6b-40aa-9c73-f64219b771dd","Type":"ContainerStarted","Data":"a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15"}
Oct 30 21:59:53.140720 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.140611    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 21:59:53.140720 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.140636    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" event={"ID":"d1820885-4e6b-40aa-9c73-f64219b771dd","Type":"ContainerStarted","Data":"aed17c76a93975d33c66e20d06ed26b38e104b6ad76f1d320a65ced752ea188a"}
Oct 30 21:59:53.259608 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.259558    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" podStartSLOduration=2.259543518 podStartE2EDuration="2.259543518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:53.258566105 +0000 UTC m=+604.782708869" watchObservedRunningTime="2026-10-30 21:59:53.259543518 +0000 UTC m=+604.783686281"
Oct 30 21:59:53.381736 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.381685    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" podStartSLOduration=2.381670701 podStartE2EDuration="2.381670701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:53.379800102 +0000 UTC m=+604.903942861" watchObservedRunningTime="2026-10-30 21:59:53.381670701 +0000 UTC m=+604.905813464"
Oct 30 21:59:53.384368 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.384342    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:53.611507 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.611478    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:53.847344 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:53.847303    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:54.080990 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.080900    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:54.202740 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.202698    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"]
Oct 30 21:59:54.206570 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.206551    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.261092 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.261061    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.261302 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.261244    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zbzc\" (UniqueName: \"kubernetes.io/projected/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-kube-api-access-9zbzc\") pod \"payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.264336 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.264314    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d8837a-dockercfg-nk7cg\""
Oct 30 21:59:54.265670 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.265652    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d8837a\""
Oct 30 21:59:54.266372 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.266350    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"]
Oct 30 21:59:54.353282 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.353251    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"]
Oct 30 21:59:54.356967 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.356950    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.362690 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.362664    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zbzc\" (UniqueName: \"kubernetes.io/projected/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-kube-api-access-9zbzc\") pod \"payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.362825 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.362807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.363659 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.363492    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.425379 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.425341    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"]
Oct 30 21:59:54.427960 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.427926    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zbzc\" (UniqueName: \"kubernetes.io/projected/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-kube-api-access-9zbzc\") pod \"payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.463465 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.463394    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfq9\" (UniqueName: \"kubernetes.io/projected/43c64e5b-6293-4414-b81d-9711ddda20fa-kube-api-access-lsfq9\") pod \"payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.463613 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.463532    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c64e5b-6293-4414-b81d-9711ddda20fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.520319 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.520282    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:54.564192 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.564156    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c64e5b-6293-4414-b81d-9711ddda20fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.564374 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.564258    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfq9\" (UniqueName: \"kubernetes.io/projected/43c64e5b-6293-4414-b81d-9711ddda20fa-kube-api-access-lsfq9\") pod \"payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.564847 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.564829    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c64e5b-6293-4414-b81d-9711ddda20fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.628996 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.628928    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfq9\" (UniqueName: \"kubernetes.io/projected/43c64e5b-6293-4414-b81d-9711ddda20fa-kube-api-access-lsfq9\") pod \"payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.669902 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.669876    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:54.896684 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:54.896657    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"]
Oct 30 21:59:54.897272 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:54.897241    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a56ce6_6e63_4314_a6c6_fef93d6e0fda.slice/crio-5fb7c975a96832a4451e34cc6dcf5ce340df599ba2f9c69e0245aca318707998 WatchSource:0}: Error finding container 5fb7c975a96832a4451e34cc6dcf5ce340df599ba2f9c69e0245aca318707998: Status 404 returned error can't find the container with id 5fb7c975a96832a4451e34cc6dcf5ce340df599ba2f9c69e0245aca318707998
Oct 30 21:59:55.047533 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.047509    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"]
Oct 30 21:59:55.048304 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:55.048279    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c64e5b_6293_4414_b81d_9711ddda20fa.slice/crio-03fb8e6d9c9ec2db26836c76296e6fcfe37a1339dec3be643c7cfaf2332f5343 WatchSource:0}: Error finding container 03fb8e6d9c9ec2db26836c76296e6fcfe37a1339dec3be643c7cfaf2332f5343: Status 404 returned error can't find the container with id 03fb8e6d9c9ec2db26836c76296e6fcfe37a1339dec3be643c7cfaf2332f5343
Oct 30 21:59:55.154431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.154329    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" event={"ID":"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda","Type":"ContainerStarted","Data":"bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982"}
Oct 30 21:59:55.154431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.154403    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 21:59:55.154431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.154428    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" event={"ID":"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda","Type":"ContainerStarted","Data":"5fb7c975a96832a4451e34cc6dcf5ce340df599ba2f9c69e0245aca318707998"}
Oct 30 21:59:55.155986 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.155963    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" event={"ID":"43c64e5b-6293-4414-b81d-9711ddda20fa","Type":"ContainerStarted","Data":"386de39497ff07e183c69d5a1a72a2a97255f976cc463a7ec2c588ecf44c9a16"}
Oct 30 21:59:55.156085 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.156009    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 21:59:55.156085 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.156026    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" event={"ID":"43c64e5b-6293-4414-b81d-9711ddda20fa","Type":"ContainerStarted","Data":"03fb8e6d9c9ec2db26836c76296e6fcfe37a1339dec3be643c7cfaf2332f5343"}
Oct 30 21:59:55.276112 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.276062    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" podStartSLOduration=1.276048464 podStartE2EDuration="1.276048464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:55.27529111 +0000 UTC m=+606.799433891" watchObservedRunningTime="2026-10-30 21:59:55.276048464 +0000 UTC m=+606.800191389"
Oct 30 21:59:55.402431 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:55.402380    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" podStartSLOduration=1.402367345 podStartE2EDuration="1.402367345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:55.401064294 +0000 UTC m=+606.925207062" watchObservedRunningTime="2026-10-30 21:59:55.402367345 +0000 UTC m=+606.926510155"
Oct 30 21:59:56.404999 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.404969    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:56.661878 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.661760    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:56.911470 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.911438    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"]
Oct 30 21:59:56.915425 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.915363    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:56.941167 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.941116    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:56.976420 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.976396    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"]
Oct 30 21:59:56.991587 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.991557    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12856bb0-d938-4e9e-81a4-84020b390535-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:56.991717 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:56.991610    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxv6g\" (UniqueName: \"kubernetes.io/projected/12856bb0-d938-4e9e-81a4-84020b390535-kube-api-access-pxv6g\") pod \"payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:57.023137 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.023109    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8ca52a\""
Oct 30 21:59:57.044587 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.044559    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"]
Oct 30 21:59:57.048421 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.048405    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.095254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.092792    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213406d0-a791-4e30-bd71-df34dad889ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.095254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.092991    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12856bb0-d938-4e9e-81a4-84020b390535-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:57.095254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.093055    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxv6g\" (UniqueName: \"kubernetes.io/projected/12856bb0-d938-4e9e-81a4-84020b390535-kube-api-access-pxv6g\") pod \"payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:57.095254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.093164    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh65t\" (UniqueName: \"kubernetes.io/projected/213406d0-a791-4e30-bd71-df34dad889ae-kube-api-access-qh65t\") pod \"payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.095254 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.094141    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12856bb0-d938-4e9e-81a4-84020b390535-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:57.104960 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.104935    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8ca52a-dockercfg-5xd2s\""
Oct 30 21:59:57.107883 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.107859    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"]
Oct 30 21:59:57.156957 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.156923    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxv6g\" (UniqueName: \"kubernetes.io/projected/12856bb0-d938-4e9e-81a4-84020b390535-kube-api-access-pxv6g\") pod \"payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:57.194618 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.194536    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh65t\" (UniqueName: \"kubernetes.io/projected/213406d0-a791-4e30-bd71-df34dad889ae-kube-api-access-qh65t\") pod \"payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.194777 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.194659    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213406d0-a791-4e30-bd71-df34dad889ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.195321 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.195305    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213406d0-a791-4e30-bd71-df34dad889ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.229226 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.229180    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:57.259646 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.259620    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh65t\" (UniqueName: \"kubernetes.io/projected/213406d0-a791-4e30-bd71-df34dad889ae-kube-api-access-qh65t\") pod \"payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.364369 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.364333    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:57.612963 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.612940    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"]
Oct 30 21:59:57.614183 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:57.614160    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12856bb0_d938_4e9e_81a4_84020b390535.slice/crio-22957ddb8e4806942e30ed7aef21ea2c203e7ed196902ed0902c18a4e5052259 WatchSource:0}: Error finding container 22957ddb8e4806942e30ed7aef21ea2c203e7ed196902ed0902c18a4e5052259: Status 404 returned error can't find the container with id 22957ddb8e4806942e30ed7aef21ea2c203e7ed196902ed0902c18a4e5052259
Oct 30 21:59:57.755008 ip-10-0-141-147 kubenswrapper[5031]: W1030 21:59:57.754968    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213406d0_a791_4e30_bd71_df34dad889ae.slice/crio-24471e477b900dd98ded375f95f1a67b66a48c46702701046fc124b6d59fae65 WatchSource:0}: Error finding container 24471e477b900dd98ded375f95f1a67b66a48c46702701046fc124b6d59fae65: Status 404 returned error can't find the container with id 24471e477b900dd98ded375f95f1a67b66a48c46702701046fc124b6d59fae65
Oct 30 21:59:57.755492 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:57.755401    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"]
Oct 30 21:59:58.180270 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.180236    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" event={"ID":"12856bb0-d938-4e9e-81a4-84020b390535","Type":"ContainerStarted","Data":"b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4"}
Oct 30 21:59:58.180463 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.180303    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 21:59:58.180463 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.180325    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" event={"ID":"12856bb0-d938-4e9e-81a4-84020b390535","Type":"ContainerStarted","Data":"22957ddb8e4806942e30ed7aef21ea2c203e7ed196902ed0902c18a4e5052259"}
Oct 30 21:59:58.181780 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.181755    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" event={"ID":"213406d0-a791-4e30-bd71-df34dad889ae","Type":"ContainerStarted","Data":"0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7"}
Oct 30 21:59:58.181890 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.181799    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" event={"ID":"213406d0-a791-4e30-bd71-df34dad889ae","Type":"ContainerStarted","Data":"24471e477b900dd98ded375f95f1a67b66a48c46702701046fc124b6d59fae65"}
Oct 30 21:59:58.181890 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.181883    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 21:59:58.300996 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.300938    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" podStartSLOduration=2.300919547 podStartE2EDuration="2.300919547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:58.300822556 +0000 UTC m=+609.824965330" watchObservedRunningTime="2026-10-30 21:59:58.300919547 +0000 UTC m=+609.825062311"
Oct 30 21:59:58.426706 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:58.426637    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" podStartSLOduration=2.426617888 podStartE2EDuration="2.426617888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:58.421289297 +0000 UTC m=+609.945432089" watchObservedRunningTime="2026-10-30 21:59:58.426617888 +0000 UTC m=+609.950760659"
Oct 30 21:59:59.204996 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.204964    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:59.473530 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.473395    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:59.721081 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.721032    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"]
Oct 30 21:59:59.728416 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.728343    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 21:59:59.736141 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.735390    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:59.795028 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.795002    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"]
Oct 30 21:59:59.828308 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.828270    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d9c20c-be0d-4589-8695-5723c3737517-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 21:59:59.828498 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.828367    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6x7h\" (UniqueName: \"kubernetes.io/projected/e4d9c20c-be0d-4589-8695-5723c3737517-kube-api-access-t6x7h\") pod \"payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 21:59:59.841262 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.841224    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-3783c2-dockercfg-6g998\""
Oct 30 21:59:59.841373 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.841271    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-3783c2\""
Oct 30 21:59:59.929503 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.929471    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d9c20c-be0d-4589-8695-5723c3737517-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 21:59:59.929665 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.929596    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6x7h\" (UniqueName: \"kubernetes.io/projected/e4d9c20c-be0d-4589-8695-5723c3737517-kube-api-access-t6x7h\") pod \"payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 21:59:59.930128 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.930107    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d9c20c-be0d-4589-8695-5723c3737517-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 21:59:59.998855 ip-10-0-141-147 kubenswrapper[5031]: I1030 21:59:59.998784    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6x7h\" (UniqueName: \"kubernetes.io/projected/e4d9c20c-be0d-4589-8695-5723c3737517-kube-api-access-t6x7h\") pod \"payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 22:00:00.045917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.045886    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 22:00:00.206674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.206642    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"]
Oct 30 22:00:00.210952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.210928    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.212502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.212480    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"]
Oct 30 22:00:00.216045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.216027    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.232955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.232931    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d53bb2e-b4d4-4c01-944c-9af481d7184a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-df72z\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.233138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.233122    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx466\" (UniqueName: \"kubernetes.io/projected/7d53bb2e-b4d4-4c01-944c-9af481d7184a-kube-api-access-dx466\") pod \"maas-api-key-cleanup-29889960-df72z\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.269467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.269340    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:00:00.269869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.269845    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:00.272493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.272452    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"]
Oct 30 22:00:00.275998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.275974    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"]
Oct 30 22:00:00.334451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.334421    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx466\" (UniqueName: \"kubernetes.io/projected/7d53bb2e-b4d4-4c01-944c-9af481d7184a-kube-api-access-dx466\") pod \"maas-api-key-cleanup-29889960-df72z\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.334652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.334466    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d53bb2e-b4d4-4c01-944c-9af481d7184a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-df72z\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.334717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.334640    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkfs\" (UniqueName: \"kubernetes.io/projected/57748a4b-7364-4cef-926b-1bf94d273447-kube-api-access-hhkfs\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.334866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.334841    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57748a4b-7364-4cef-926b-1bf94d273447-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.335139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.335065    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d53bb2e-b4d4-4c01-944c-9af481d7184a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-df72z\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.370665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.370636    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:00.397892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.397866    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx466\" (UniqueName: \"kubernetes.io/projected/7d53bb2e-b4d4-4c01-944c-9af481d7184a-kube-api-access-dx466\") pod \"maas-api-key-cleanup-29889960-df72z\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.424074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.424048    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"]
Oct 30 22:00:00.425274 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:00:00.425250    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d9c20c_be0d_4589_8695_5723c3737517.slice/crio-f94209dbc0c86daf5a71b428728a690d8c04bb11d9f5cbd467f2e5f88c8c1980 WatchSource:0}: Error finding container f94209dbc0c86daf5a71b428728a690d8c04bb11d9f5cbd467f2e5f88c8c1980: Status 404 returned error can't find the container with id f94209dbc0c86daf5a71b428728a690d8c04bb11d9f5cbd467f2e5f88c8c1980
Oct 30 22:00:00.436179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.436155    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkfs\" (UniqueName: \"kubernetes.io/projected/57748a4b-7364-4cef-926b-1bf94d273447-kube-api-access-hhkfs\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.436283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.436246    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57748a4b-7364-4cef-926b-1bf94d273447-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.436817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.436800    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57748a4b-7364-4cef-926b-1bf94d273447-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.499970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.499941    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkfs\" (UniqueName: \"kubernetes.io/projected/57748a4b-7364-4cef-926b-1bf94d273447-kube-api-access-hhkfs\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.526756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.526682    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:00.532685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.532665    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:00.925847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.925824    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"]
Oct 30 22:00:00.926538 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:00:00.926511    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57748a4b_7364_4cef_926b_1bf94d273447.slice/crio-7e773a89a6f66529c67d47487cc6333b53651593a5a4f3592df3f4527b299bf4 WatchSource:0}: Error finding container 7e773a89a6f66529c67d47487cc6333b53651593a5a4f3592df3f4527b299bf4: Status 404 returned error can't find the container with id 7e773a89a6f66529c67d47487cc6333b53651593a5a4f3592df3f4527b299bf4
Oct 30 22:00:00.973448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:00.973418    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"]
Oct 30 22:00:00.973858 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:00:00.973835    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d53bb2e_b4d4_4c01_944c_9af481d7184a.slice/crio-e302037141a36fef7beb6660901618e8a664613b0d190fd0bc96091d0104c0aa WatchSource:0}: Error finding container e302037141a36fef7beb6660901618e8a664613b0d190fd0bc96091d0104c0aa: Status 404 returned error can't find the container with id e302037141a36fef7beb6660901618e8a664613b0d190fd0bc96091d0104c0aa
Oct 30 22:00:01.202806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:01.202710    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerStarted","Data":"7e773a89a6f66529c67d47487cc6333b53651593a5a4f3592df3f4527b299bf4"}
Oct 30 22:00:01.204371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:01.204339    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" event={"ID":"e4d9c20c-be0d-4589-8695-5723c3737517","Type":"ContainerStarted","Data":"69c91ad8ed84fce2d73e62eeb42665cb050c3756bb3f4648c20f6571f325083d"}
Oct 30 22:00:01.204503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:01.204401    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 22:00:01.204503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:01.204425    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" event={"ID":"e4d9c20c-be0d-4589-8695-5723c3737517","Type":"ContainerStarted","Data":"f94209dbc0c86daf5a71b428728a690d8c04bb11d9f5cbd467f2e5f88c8c1980"}
Oct 30 22:00:01.205721 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:01.205691    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerStarted","Data":"e302037141a36fef7beb6660901618e8a664613b0d190fd0bc96091d0104c0aa"}
Oct 30 22:00:01.330071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:01.330000    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" podStartSLOduration=2.329983955 podStartE2EDuration="2.329983955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.328196119 +0000 UTC m=+612.852338901" watchObservedRunningTime="2026-10-30 22:00:01.329983955 +0000 UTC m=+612.854126758"
Oct 30 22:00:01.980143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:01.980107    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:00:02.051732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:02.051701    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:02.278038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:02.277957    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:04.149566 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:04.149505    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 22:00:04.150137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:04.149838    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 22:00:04.821708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:04.821669    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:05.067258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.063843    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:05.274257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.274226    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"]
Oct 30 22:00:05.281782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.281756    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:05.299645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.298823    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:05.386814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.386781    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-bce513-dockercfg-bvrlx\""
Oct 30 22:00:05.390901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.390875    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"]
Oct 30 22:00:05.391067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.390983    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-bce513\""
Oct 30 22:00:05.396873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.396621    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:05.396873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.396805    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4fg\" (UniqueName: \"kubernetes.io/projected/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-kube-api-access-nz4fg\") pod \"payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:05.498793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.498466    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4fg\" (UniqueName: \"kubernetes.io/projected/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-kube-api-access-nz4fg\") pod \"payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:05.498793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.498592    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:05.499389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.499365    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:05.586178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.586104    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4fg\" (UniqueName: \"kubernetes.io/projected/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-kube-api-access-nz4fg\") pod \"payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:05.620773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:05.620580    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:06.037978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.037948    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"]
Oct 30 22:00:06.108937 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:00:06.108902    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730e9a61_0bf3_44bf_bd86_3cad576ab1d0.slice/crio-0a3c6dc074d212d4b687c205f487ec72eae47a4c00965982ca0bd51f78df04d5 WatchSource:0}: Error finding container 0a3c6dc074d212d4b687c205f487ec72eae47a4c00965982ca0bd51f78df04d5: Status 404 returned error can't find the container with id 0a3c6dc074d212d4b687c205f487ec72eae47a4c00965982ca0bd51f78df04d5
Oct 30 22:00:06.168586 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.168562    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 22:00:06.168715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.168619    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 22:00:06.243455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.243422    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerStarted","Data":"bfbbf8d462ebb9a45e3d6b6ffe210f4ec3258b11321b64cd98a7bd9da9a250f9"}
Oct 30 22:00:06.244954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.244931    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" event={"ID":"730e9a61-0bf3-44bf-bd86-3cad576ab1d0","Type":"ContainerStarted","Data":"4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9"}
Oct 30 22:00:06.245081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.244973    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" event={"ID":"730e9a61-0bf3-44bf-bd86-3cad576ab1d0","Type":"ContainerStarted","Data":"0a3c6dc074d212d4b687c205f487ec72eae47a4c00965982ca0bd51f78df04d5"}
Oct 30 22:00:06.245081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.245002    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:06.246515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.246497    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerStarted","Data":"0ad165ea64eda7f863fae3d992deace4a7b03817cdf6330454f53363089549dd"}
Oct 30 22:00:06.564408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.564358    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" podStartSLOduration=1.8916308530000001 podStartE2EDuration="6.564342114s" totalImagesPullingTime="4.672711261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:06.561575247 +0000 UTC m=+618.085718009" watchObservedRunningTime="2026-10-30 22:00:06.564342114 +0000 UTC m=+618.088484913"
Oct 30 22:00:06.686028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.685930    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" podStartSLOduration=1.685912873 podStartE2EDuration="1.685912873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:06.684557404 +0000 UTC m=+618.208700168" watchObservedRunningTime="2026-10-30 22:00:06.685912873 +0000 UTC m=+618.210055636"
Oct 30 22:00:06.811082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:06.811002    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" podStartSLOduration=1.89774749 podStartE2EDuration="6.810982361s" totalImagesPullingTime="4.913234871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:06.806983711 +0000 UTC m=+618.331126484" watchObservedRunningTime="2026-10-30 22:00:06.810982361 +0000 UTC m=+618.335125125"
Oct 30 22:00:07.255591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.255554    5031 generic.go:356] "Generic (PLEG): container finished" podID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerID="bfbbf8d462ebb9a45e3d6b6ffe210f4ec3258b11321b64cd98a7bd9da9a250f9" exitCode=60
Oct 30 22:00:07.255793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.255690    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerDied","Data":"bfbbf8d462ebb9a45e3d6b6ffe210f4ec3258b11321b64cd98a7bd9da9a250f9"}
Oct 30 22:00:07.255981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.255959    5031 scope.go:122] "RemoveContainer" containerID="bfbbf8d462ebb9a45e3d6b6ffe210f4ec3258b11321b64cd98a7bd9da9a250f9"
Oct 30 22:00:07.257593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.257557    5031 generic.go:356] "Generic (PLEG): container finished" podID="57748a4b-7364-4cef-926b-1bf94d273447" containerID="0ad165ea64eda7f863fae3d992deace4a7b03817cdf6330454f53363089549dd" exitCode=7
Oct 30 22:00:07.257700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.257672    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerDied","Data":"0ad165ea64eda7f863fae3d992deace4a7b03817cdf6330454f53363089549dd"}
Oct 30 22:00:07.257905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.257883    5031 scope.go:122] "RemoveContainer" containerID="0ad165ea64eda7f863fae3d992deace4a7b03817cdf6330454f53363089549dd"
Oct 30 22:00:07.587309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.587264    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:07.851890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.851852    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:07.942100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.942064    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"]
Oct 30 22:00:07.950235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:07.950177    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.006329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.006297    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"]
Oct 30 22:00:08.014316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.014291    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-9cff60-dockercfg-pllnt\""
Oct 30 22:00:08.014641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.014624    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-9cff60\""
Oct 30 22:00:08.065064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.063920    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7p5\" (UniqueName: \"kubernetes.io/projected/2426ab6c-129f-4da4-b968-f508ceab1021-kube-api-access-8k7p5\") pod \"payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.065064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.064012    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426ab6c-129f-4da4-b968-f508ceab1021-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.164912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.164820    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7p5\" (UniqueName: \"kubernetes.io/projected/2426ab6c-129f-4da4-b968-f508ceab1021-kube-api-access-8k7p5\") pod \"payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.164912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.164880    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426ab6c-129f-4da4-b968-f508ceab1021-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.165701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.165674    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426ab6c-129f-4da4-b968-f508ceab1021-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.231446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.231424    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7p5\" (UniqueName: \"kubernetes.io/projected/2426ab6c-129f-4da4-b968-f508ceab1021-kube-api-access-8k7p5\") pod \"payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.265529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.265475    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:08.271367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.271338    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerStarted","Data":"3468b94f7aeb386313221da7d96d1f1c8033243b47c131b6bd1f6b657ed7acc4"}
Oct 30 22:00:08.274251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.274222    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerStarted","Data":"60dd3e5d511887d8429a194133a31a5a2e9092f5a70d726769e652a0bc88a9ec"}
Oct 30 22:00:08.697330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:08.697306    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"]
Oct 30 22:00:08.699136 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:00:08.699105    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2426ab6c_129f_4da4_b968_f508ceab1021.slice/crio-459395b48a825e35c598ff145275b4c79ad6639d39476bdf94c23341e398bed2 WatchSource:0}: Error finding container 459395b48a825e35c598ff145275b4c79ad6639d39476bdf94c23341e398bed2: Status 404 returned error can't find the container with id 459395b48a825e35c598ff145275b4c79ad6639d39476bdf94c23341e398bed2
Oct 30 22:00:09.192709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.192680    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 22:00:09.192929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.192914    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 22:00:09.282516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.282484    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" event={"ID":"2426ab6c-129f-4da4-b968-f508ceab1021","Type":"ContainerStarted","Data":"42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1"}
Oct 30 22:00:09.282706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.282539    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:09.282706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.282560    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" event={"ID":"2426ab6c-129f-4da4-b968-f508ceab1021","Type":"ContainerStarted","Data":"459395b48a825e35c598ff145275b4c79ad6639d39476bdf94c23341e398bed2"}
Oct 30 22:00:09.284469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.284444    5031 generic.go:356] "Generic (PLEG): container finished" podID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerID="3468b94f7aeb386313221da7d96d1f1c8033243b47c131b6bd1f6b657ed7acc4" exitCode=60
Oct 30 22:00:09.284582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.284524    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerDied","Data":"3468b94f7aeb386313221da7d96d1f1c8033243b47c131b6bd1f6b657ed7acc4"}
Oct 30 22:00:09.284582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.284575    5031 scope.go:122] "RemoveContainer" containerID="bfbbf8d462ebb9a45e3d6b6ffe210f4ec3258b11321b64cd98a7bd9da9a250f9"
Oct 30 22:00:09.284701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.284685    5031 scope.go:122] "RemoveContainer" containerID="3468b94f7aeb386313221da7d96d1f1c8033243b47c131b6bd1f6b657ed7acc4"
Oct 30 22:00:09.284945 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:09.284922    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-df72z_odh-ai-gateway-infra(7d53bb2e-b4d4-4c01-944c-9af481d7184a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a"
Oct 30 22:00:09.286490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.286454    5031 generic.go:356] "Generic (PLEG): container finished" podID="57748a4b-7364-4cef-926b-1bf94d273447" containerID="60dd3e5d511887d8429a194133a31a5a2e9092f5a70d726769e652a0bc88a9ec" exitCode=60
Oct 30 22:00:09.286600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.286534    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerDied","Data":"60dd3e5d511887d8429a194133a31a5a2e9092f5a70d726769e652a0bc88a9ec"}
Oct 30 22:00:09.286722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.286708    5031 scope.go:122] "RemoveContainer" containerID="60dd3e5d511887d8429a194133a31a5a2e9092f5a70d726769e652a0bc88a9ec"
Oct 30 22:00:09.286904 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:09.286888    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb_odh-ai-gateway-infra(57748a4b-7364-4cef-926b-1bf94d273447)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" podUID="57748a4b-7364-4cef-926b-1bf94d273447"
Oct 30 22:00:09.297487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.297471    5031 scope.go:122] "RemoveContainer" containerID="0ad165ea64eda7f863fae3d992deace4a7b03817cdf6330454f53363089549dd"
Oct 30 22:00:09.459983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:09.459868    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" podStartSLOduration=2.459841664 podStartE2EDuration="2.459841664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.458426505 +0000 UTC m=+620.982569286" watchObservedRunningTime="2026-10-30 22:00:09.459841664 +0000 UTC m=+620.983984426"
Oct 30 22:00:10.295430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:10.295404    5031 scope.go:122] "RemoveContainer" containerID="3468b94f7aeb386313221da7d96d1f1c8033243b47c131b6bd1f6b657ed7acc4"
Oct 30 22:00:10.295858 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:10.295685    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-df72z_odh-ai-gateway-infra(7d53bb2e-b4d4-4c01-944c-9af481d7184a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a"
Oct 30 22:00:10.297020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:10.297003    5031 scope.go:122] "RemoveContainer" containerID="60dd3e5d511887d8429a194133a31a5a2e9092f5a70d726769e652a0bc88a9ec"
Oct 30 22:00:10.297252 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:10.297195    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb_odh-ai-gateway-infra(57748a4b-7364-4cef-926b-1bf94d273447)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" podUID="57748a4b-7364-4cef-926b-1bf94d273447"
Oct 30 22:00:11.421319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:11.421279    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:11.587215 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:11.587173    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:00:12.215159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:12.215133    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 22:00:17.260942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:17.260916    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:00:17.451622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:17.451579    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:17.808116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:17.808086    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:18.120249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:18.120200    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:20.048233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:20.047598    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:20.300713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:20.300642    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:00:20.540998 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:20.540954    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:00:21.395768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.395737    5031 generic.go:356] "Generic (PLEG): container finished" podID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerID="4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3" exitCode=0
Oct 30 22:00:21.396256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.395813    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" event={"ID":"646d59da-3bfd-4e6a-a6b1-5e387db1bb44","Type":"ContainerDied","Data":"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"}
Oct 30 22:00:21.396256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.396183    5031 scope.go:122] "RemoveContainer" containerID="4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"
Oct 30 22:00:21.397609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.397573    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:21.397744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.397661    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:21.397744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.397711    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:21.638506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.638479    5031 scope.go:122] "RemoveContainer" containerID="3468b94f7aeb386313221da7d96d1f1c8033243b47c131b6bd1f6b657ed7acc4"
Oct 30 22:00:21.638683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:21.638666    5031 scope.go:122] "RemoveContainer" containerID="60dd3e5d511887d8429a194133a31a5a2e9092f5a70d726769e652a0bc88a9ec"
Oct 30 22:00:22.404532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:22.404498    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerStarted","Data":"23b8e17976e608e217726aaa596b4450750c2b3531417bbb6e327481cf6bbf9b"}
Oct 30 22:00:22.433250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:22.426171    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" event={"ID":"646d59da-3bfd-4e6a-a6b1-5e387db1bb44","Type":"ContainerStarted","Data":"f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c"}
Oct 30 22:00:22.444613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:22.444572    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerStarted","Data":"9db5320669cf2ff06d66f3391c954c4aa6cc282a845ad9b7a1ad61721115c06c"}
Oct 30 22:00:22.882957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:22.882921    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 22:00:22.883152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:22.883129    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:00:22.883227 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:22.883191    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:00:23.436952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.436913    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:23.459231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.459168    5031 generic.go:356] "Generic (PLEG): container finished" podID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerID="9db5320669cf2ff06d66f3391c954c4aa6cc282a845ad9b7a1ad61721115c06c" exitCode=60
Oct 30 22:00:23.459416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.459246    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerDied","Data":"9db5320669cf2ff06d66f3391c954c4aa6cc282a845ad9b7a1ad61721115c06c"}
Oct 30 22:00:23.459416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.459346    5031 scope.go:122] "RemoveContainer" containerID="3468b94f7aeb386313221da7d96d1f1c8033243b47c131b6bd1f6b657ed7acc4"
Oct 30 22:00:23.459689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.459673    5031 scope.go:122] "RemoveContainer" containerID="9db5320669cf2ff06d66f3391c954c4aa6cc282a845ad9b7a1ad61721115c06c"
Oct 30 22:00:23.460103 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:23.459928    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-df72z_odh-ai-gateway-infra(7d53bb2e-b4d4-4c01-944c-9af481d7184a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a"
Oct 30 22:00:23.461512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.461492    5031 generic.go:356] "Generic (PLEG): container finished" podID="57748a4b-7364-4cef-926b-1bf94d273447" containerID="23b8e17976e608e217726aaa596b4450750c2b3531417bbb6e327481cf6bbf9b" exitCode=60
Oct 30 22:00:23.461630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.461565    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerDied","Data":"23b8e17976e608e217726aaa596b4450750c2b3531417bbb6e327481cf6bbf9b"}
Oct 30 22:00:23.461918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.461898    5031 scope.go:122] "RemoveContainer" containerID="23b8e17976e608e217726aaa596b4450750c2b3531417bbb6e327481cf6bbf9b"
Oct 30 22:00:23.462158 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:23.462137    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb_odh-ai-gateway-infra(57748a4b-7364-4cef-926b-1bf94d273447)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" podUID="57748a4b-7364-4cef-926b-1bf94d273447"
Oct 30 22:00:23.474574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.474555    5031 scope.go:122] "RemoveContainer" containerID="60dd3e5d511887d8429a194133a31a5a2e9092f5a70d726769e652a0bc88a9ec"
Oct 30 22:00:23.559122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.559088    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"]
Oct 30 22:00:23.805174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.805090    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:23.834235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.834182    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"]
Oct 30 22:00:23.883030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.882997    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:00:23.883197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:23.883055    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:00:24.062877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.062795    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:24.752404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.752375    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:24.762326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.762304    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:24.774462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.774431    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57748a4b-7364-4cef-926b-1bf94d273447-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57748a4b-7364-4cef-926b-1bf94d273447-maas-api-service-ca\") pod \"57748a4b-7364-4cef-926b-1bf94d273447\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") "
Oct 30 22:00:24.774591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.774506    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57748a4b-7364-4cef-926b-1bf94d273447-kube-api-access-hhkfs\" (UniqueName: \"kubernetes.io/projected/57748a4b-7364-4cef-926b-1bf94d273447-kube-api-access-hhkfs\") pod \"57748a4b-7364-4cef-926b-1bf94d273447\" (UID: \"57748a4b-7364-4cef-926b-1bf94d273447\") "
Oct 30 22:00:24.774829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.774808    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57748a4b-7364-4cef-926b-1bf94d273447-maas-api-service-ca" pod "57748a4b-7364-4cef-926b-1bf94d273447" (UID: "57748a4b-7364-4cef-926b-1bf94d273447"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:24.774926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.774912    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57748a4b-7364-4cef-926b-1bf94d273447-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:24.776868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.776847    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57748a4b-7364-4cef-926b-1bf94d273447-kube-api-access-hhkfs" pod "57748a4b-7364-4cef-926b-1bf94d273447" (UID: "57748a4b-7364-4cef-926b-1bf94d273447"). InnerVolumeSpecName "kube-api-access-hhkfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:24.876124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.876090    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d53bb2e-b4d4-4c01-944c-9af481d7184a-kube-api-access-dx466\" (UniqueName: \"kubernetes.io/projected/7d53bb2e-b4d4-4c01-944c-9af481d7184a-kube-api-access-dx466\") pod \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") "
Oct 30 22:00:24.876335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.876166    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d53bb2e-b4d4-4c01-944c-9af481d7184a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d53bb2e-b4d4-4c01-944c-9af481d7184a-maas-api-service-ca\") pod \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\" (UID: \"7d53bb2e-b4d4-4c01-944c-9af481d7184a\") "
Oct 30 22:00:24.876424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.876412    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkfs\" (UniqueName: \"kubernetes.io/projected/57748a4b-7364-4cef-926b-1bf94d273447-kube-api-access-hhkfs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:24.876499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.876479    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d53bb2e-b4d4-4c01-944c-9af481d7184a-maas-api-service-ca" pod "7d53bb2e-b4d4-4c01-944c-9af481d7184a" (UID: "7d53bb2e-b4d4-4c01-944c-9af481d7184a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:24.878336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.878316    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d53bb2e-b4d4-4c01-944c-9af481d7184a-kube-api-access-dx466" pod "7d53bb2e-b4d4-4c01-944c-9af481d7184a" (UID: "7d53bb2e-b4d4-4c01-944c-9af481d7184a"). InnerVolumeSpecName "kube-api-access-dx466". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:24.882713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.882689    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:00:24.882813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.882757    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:00:24.977333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.977302    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx466\" (UniqueName: \"kubernetes.io/projected/7d53bb2e-b4d4-4c01-944c-9af481d7184a-kube-api-access-dx466\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:24.977333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:24.977328    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d53bb2e-b4d4-4c01-944c-9af481d7184a-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:25.482875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.482838    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb" event={"ID":"57748a4b-7364-4cef-926b-1bf94d273447","Type":"ContainerDied","Data":"7e773a89a6f66529c67d47487cc6333b53651593a5a4f3592df3f4527b299bf4"}
Oct 30 22:00:25.483049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.482909    5031 scope.go:122] "RemoveContainer" containerID="23b8e17976e608e217726aaa596b4450750c2b3531417bbb6e327481cf6bbf9b"
Oct 30 22:00:25.483049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.482994    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"
Oct 30 22:00:25.494410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.494380    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z" event={"ID":"7d53bb2e-b4d4-4c01-944c-9af481d7184a","Type":"ContainerDied","Data":"e302037141a36fef7beb6660901618e8a664613b0d190fd0bc96091d0104c0aa"}
Oct 30 22:00:25.494525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.494483    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"
Oct 30 22:00:25.503234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.503197    5031 scope.go:122] "RemoveContainer" containerID="9db5320669cf2ff06d66f3391c954c4aa6cc282a845ad9b7a1ad61721115c06c"
Oct 30 22:00:25.678067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.677805    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"]
Oct 30 22:00:25.736291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.735228    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889960-bdclb"]
Oct 30 22:00:25.799491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.799458    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"]
Oct 30 22:00:25.805630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.805609    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-df72z"]
Oct 30 22:00:25.882660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.882629    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:00:25.882825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:25.882686    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:00:26.632024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:26.631973    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:00:26.645879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:26.645850    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57748a4b-7364-4cef-926b-1bf94d273447" path="/var/lib/kubelet/pods/57748a4b-7364-4cef-926b-1bf94d273447/volumes"
Oct 30 22:00:26.646254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:26.646241    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" path="/var/lib/kubelet/pods/7d53bb2e-b4d4-4c01-944c-9af481d7184a/volumes"
Oct 30 22:00:26.883000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:26.882924    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:00:26.883000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:26.882979    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:00:27.886782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:27.886741    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 22:00:27.887320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:27.887006    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 22:00:27.887669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:27.887652    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 22:00:29.554384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:29.554353    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:34.724691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:34.724652    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:34.978275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:34.978189    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:35.644125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:35.644092    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:00:36.597643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:36.597608    5031 generic.go:356] "Generic (PLEG): container finished" podID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerID="5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f" exitCode=0
Oct 30 22:00:36.598061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:36.597680    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" event={"ID":"ac3999f7-0b9d-4cb8-8023-e1703c40c68b","Type":"ContainerDied","Data":"5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f"}
Oct 30 22:00:36.598061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:36.598043    5031 scope.go:122] "RemoveContainer" containerID="5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f"
Oct 30 22:00:36.599335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:36.599303    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:36.599428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:36.599370    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:36.599428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:36.599403    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:00:37.637772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:37.637733    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" event={"ID":"ac3999f7-0b9d-4cb8-8023-e1703c40c68b","Type":"ContainerStarted","Data":"8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01"}
Oct 30 22:00:37.778743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:37.778710    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:38.191796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:38.191756    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 22:00:38.192002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:38.191976    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:00:38.192086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:38.192045    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:00:38.378509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:38.378469    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:38.635219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:38.635158    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:39.191655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:39.191624    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:00:39.192072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:39.191670    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:00:40.192508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:40.192469    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:00:40.192916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:40.192521    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:00:40.394989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:40.394959    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:40.641343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:40.641305    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:41.191730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:41.191695    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:00:41.191914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:41.191754    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:00:42.192463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.192431    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:00:42.192921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.192495    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:00:42.283556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.283518    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:42.486911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.486820    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"]
Oct 30 22:00:42.487406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487391    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487407    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487424    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487429    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487445    5031 cpu_manager.go:414] "containerMap: removing container" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487449    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487486    5031 cpu_manager.go:414] "containerMap: removing container" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487491    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487505    5031 cpu_manager.go:414] "containerMap: removing container" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487510    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487599    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487609    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:00:42.487890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487615    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:00:42.487890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487623    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:00:42.487890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.487628    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:00:42.494242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.494194    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.560100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.560073    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"kube-root-ca.crt\""
Oct 30 22:00:42.560327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.560135    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"openshift-service-ca.crt\""
Oct 30 22:00:42.603822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.603786    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-home\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.603822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.603824    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-tmp-dir\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.604032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.603933    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-model-cache\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.604032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.604005    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgrt\" (UniqueName: \"kubernetes.io/projected/16200e83-d826-4272-95ea-a6d41611f90e-kube-api-access-vvgrt\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.604111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.604088    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16200e83-d826-4272-95ea-a6d41611f90e-tls-certs\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.604111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.604106    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-dshm\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.610759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.610721    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"simulator-w2-d8837a-kserve-self-signed-certs\""
Oct 30 22:00:42.610911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.610785    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"default-dockercfg-47zx6\""
Oct 30 22:00:42.613787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.613765    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"]
Oct 30 22:00:42.705549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.705513    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-home\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.705723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.705557    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-tmp-dir\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.705723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.705605    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-model-cache\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.705723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.705702    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgrt\" (UniqueName: \"kubernetes.io/projected/16200e83-d826-4272-95ea-a6d41611f90e-kube-api-access-vvgrt\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.705919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.705903    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-home\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.706034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.706010    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-tmp-dir\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.706034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.706024    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-model-cache\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.706161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.706081    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16200e83-d826-4272-95ea-a6d41611f90e-tls-certs\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.706161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.706102    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-dshm\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.708765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.708739    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-dshm\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.709019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.709002    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16200e83-d826-4272-95ea-a6d41611f90e-tls-certs\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.741620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.741554    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:42.772900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.772876    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgrt\" (UniqueName: \"kubernetes.io/projected/16200e83-d826-4272-95ea-a6d41611f90e-kube-api-access-vvgrt\") pod \"simulator-w2-d8837a-kserve-5bd867b954-fkcvx\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") " pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:42.810915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:42.810884    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:43.196149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.196119    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 22:00:43.196682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.196645    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 22:00:43.197298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.197280    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 22:00:43.204533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.204512    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"]
Oct 30 22:00:43.205710 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:00:43.205689    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16200e83_d826_4272_95ea_a6d41611f90e.slice/crio-18354bd1d55a9f702d5aa94b8bb650499bfd22499a46b53e4a957207d41a7474 WatchSource:0}: Error finding container 18354bd1d55a9f702d5aa94b8bb650499bfd22499a46b53e4a957207d41a7474: Status 404 returned error can't find the container with id 18354bd1d55a9f702d5aa94b8bb650499bfd22499a46b53e4a957207d41a7474
Oct 30 22:00:43.262987 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.262950    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:43.689628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.689594    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" event={"ID":"16200e83-d826-4272-95ea-a6d41611f90e","Type":"ContainerStarted","Data":"263b81580bd43fb53c027282a8917f7a1332572e0941ae24eb06121fbb8c7e97"}
Oct 30 22:00:43.689823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.689640    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" event={"ID":"16200e83-d826-4272-95ea-a6d41611f90e","Type":"ContainerStarted","Data":"18354bd1d55a9f702d5aa94b8bb650499bfd22499a46b53e4a957207d41a7474"}
Oct 30 22:00:43.811416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.810932    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:43.811698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:43.811650    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podStartSLOduration=1.811632168 podStartE2EDuration="1.811632168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:43.811310002 +0000 UTC m=+655.335452757" watchObservedRunningTime="2026-10-30 22:00:43.811632168 +0000 UTC m=+655.335774932"
Oct 30 22:00:45.848554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:45.848518    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:50.540606 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:00:50.540569    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:00:51.038087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:51.038052    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:52.811295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:52.811259    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:52.824075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:52.824047    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:53.266274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:53.266192    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:53.763466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:53.763441    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:00:53.815181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:53.815152    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:54.042449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:54.042358    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:55.754647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:55.754613    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:00:56.486103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:56.486076    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:58.533301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:58.530521    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:59.136202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:59.136167    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:59.369856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:00:59.369816    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:02.024276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:02.024175    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:05.837067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:05.837032    5031 generic.go:356] "Generic (PLEG): container finished" podID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerID="4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17" exitCode=0
Oct 30 22:01:05.837553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:05.837096    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" event={"ID":"1b6e59d2-78a1-42a9-af56-e4c01eb00496","Type":"ContainerDied","Data":"4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17"}
Oct 30 22:01:05.837553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:05.837464    5031 scope.go:122] "RemoveContainer" containerID="4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17"
Oct 30 22:01:05.838962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:05.838928    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:01:05.839098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:05.839015    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:01:05.839098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:05.839062    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:01:06.844954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:06.844916    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" event={"ID":"1b6e59d2-78a1-42a9-af56-e4c01eb00496","Type":"ContainerStarted","Data":"590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f"}
Oct 30 22:01:07.102433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:07.102350    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:07.470608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:07.470515    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:07.563053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:07.563015    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 22:01:07.563306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:07.563284    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:01:07.563355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:07.563333    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:01:07.837931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:07.837815    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:08.484569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:08.484531    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:08.563231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:08.563187    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:01:08.563423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:08.563268    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:01:08.732185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:08.732152    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:09.563613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:09.563578    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:01:09.564024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:09.563647    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:01:10.563108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:10.563076    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:01:10.563319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:10.563126    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:01:11.563821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:11.563767    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:01:11.564312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:11.563861    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 22:01:12.567889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:12.567859    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 22:01:12.568348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:12.568223    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 22:01:12.569106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:12.569090    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 22:01:12.720882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:12.720851    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:13.145942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.145910    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"]
Oct 30 22:01:13.146499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.146485    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:01:13.146551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.146501    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d53bb2e-b4d4-4c01-944c-9af481d7184a" containerName="cleanup"
Oct 30 22:01:13.146629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.146620    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="57748a4b-7364-4cef-926b-1bf94d273447" containerName="cleanup"
Oct 30 22:01:13.148780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.148766    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.207624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.207601    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d8837a-openshift-default-dockercfg-p6b8g\""
Oct 30 22:01:13.228776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.228745    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.228949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.228818    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.228949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.228894    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.229077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.229012    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.229077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.229047    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.229190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.229111    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.229190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.229161    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.229330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.229248    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llk5\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-kube-api-access-5llk5\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.229387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.229342    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.257788 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.257758    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"]
Oct 30 22:01:13.330750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.330712    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.330910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.330758    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.330910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.330887    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.330916    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.330935    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.330973    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331010    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llk5\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-kube-api-access-5llk5\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331063    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331106    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331113    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331251    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331321    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331467    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.331617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.331596    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.333456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.333433    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.333781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.333759    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.395253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.395203    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.395466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.395446    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llk5\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-kube-api-access-5llk5\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.460316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.460243    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:13.895827 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:01:13.895796    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df4d7c2_1b7b_4f23_a112_22accd419e7d.slice/crio-4c7d2681276683ac0ad721ea4a4e66ba00999f78a014a75c33497fc8c8d3ed1c WatchSource:0}: Error finding container 4c7d2681276683ac0ad721ea4a4e66ba00999f78a014a75c33497fc8c8d3ed1c: Status 404 returned error can't find the container with id 4c7d2681276683ac0ad721ea4a4e66ba00999f78a014a75c33497fc8c8d3ed1c
Oct 30 22:01:13.896178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.896155    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"]
Oct 30 22:01:13.898308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.898278    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:01:13.898411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.898336    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:01:13.898411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.898368    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:01:13.908835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:13.908805    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" event={"ID":"6df4d7c2-1b7b-4f23-a112-22accd419e7d","Type":"ContainerStarted","Data":"4c7d2681276683ac0ad721ea4a4e66ba00999f78a014a75c33497fc8c8d3ed1c"}
Oct 30 22:01:14.536114 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:14.536073    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:14.950407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:14.950364    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" event={"ID":"6df4d7c2-1b7b-4f23-a112-22accd419e7d","Type":"ContainerStarted","Data":"a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271"}
Oct 30 22:01:15.143105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:15.143062    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:15.401756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:15.401708    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:15.460471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:15.460429    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:15.462544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:15.462516    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:01:15.462690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:15.462584    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:01:16.276189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:16.276131    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podStartSLOduration=3.276114024 podStartE2EDuration="3.276114024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:15.141571503 +0000 UTC m=+686.665714294" watchObservedRunningTime="2026-10-30 22:01:16.276114024 +0000 UTC m=+687.800256789"
Oct 30 22:01:16.278393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:16.278375    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:01:16.461539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:16.461496    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:01:16.461740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:16.461566    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:01:17.461604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:17.461570    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:01:17.462137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:17.461640    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:01:18.432235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:18.431520    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:18.461393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:18.461358    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:01:18.461559 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:18.461405    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:01:19.461499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:19.461460    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:01:19.462009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:19.461536    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 22:01:19.839224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:19.839106    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:20.465906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:20.465873    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:20.466444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:20.466414    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:20.467272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:20.467253    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:01:20.541634 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:01:20.541601    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:01:23.620726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:23.620686    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:25.647752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:25.647720    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.292204 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:26.292163    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.537350 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:26.537263    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:29.013273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:29.013241    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:30.266967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:30.266912    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:30.881815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:30.881763    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:31.111719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:31.111684    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:34.398756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:34.398710    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:35.048960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:35.048922    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:01:37.200476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:37.200442    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:39.512720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:39.512678    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.667231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:42.666966    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:44.709055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:44.709016    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:45.158304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:45.158263    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:49.814426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:49.814386    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:50.542279 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:01:50.542246    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:01:52.674744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.674707    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"]
Oct 30 22:01:52.679957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.679934    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.785407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.785367    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"kube-root-ca.crt\""
Oct 30 22:01:52.785407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.785394    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"openshift-service-ca.crt\""
Oct 30 22:01:52.785696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.785679    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"simulator-w6-2182a5-kserve-self-signed-certs\""
Oct 30 22:01:52.785775 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.785729    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"default-dockercfg-kg9lm\""
Oct 30 22:01:52.788365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.788344    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"]
Oct 30 22:01:52.796320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.796296    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-dshm\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.796423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.796342    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b86265ef-a35f-4c8b-9e28-59480c6d3657-tls-certs\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.796423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.796407    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-home\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.796531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.796435    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/b86265ef-a35f-4c8b-9e28-59480c6d3657-kube-api-access-2qkq8\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.796531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.796457    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-model-cache\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.796608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.796595    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-tmp-dir\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.835372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.835339    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:52.897366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897328    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-dshm\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.897572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897383    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b86265ef-a35f-4c8b-9e28-59480c6d3657-tls-certs\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.897572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897422    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-home\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.897572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897448    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/b86265ef-a35f-4c8b-9e28-59480c6d3657-kube-api-access-2qkq8\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.897572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897474    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-model-cache\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.897572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897552    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-tmp-dir\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.897915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897893    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-home\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.897983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.897961    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-model-cache\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.898034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.898017    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-tmp-dir\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.899869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.899846    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-dshm\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.900131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.900113    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b86265ef-a35f-4c8b-9e28-59480c6d3657-tls-certs\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.962032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.961947    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/b86265ef-a35f-4c8b-9e28-59480c6d3657-kube-api-access-2qkq8\") pod \"simulator-w6-2182a5-kserve-765d949cf8-p85wp\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") " pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:52.992790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:52.992753    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:53.385259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:53.385230    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"]
Oct 30 22:01:53.385422 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:01:53.385402    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb86265ef_a35f_4c8b_9e28_59480c6d3657.slice/crio-43e8cbafd1aa17c9c2950e451c57ed40b5f0a4142bfa8c52cfbc475a13528259 WatchSource:0}: Error finding container 43e8cbafd1aa17c9c2950e451c57ed40b5f0a4142bfa8c52cfbc475a13528259: Status 404 returned error can't find the container with id 43e8cbafd1aa17c9c2950e451c57ed40b5f0a4142bfa8c52cfbc475a13528259
Oct 30 22:01:54.222055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:54.222013    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" event={"ID":"b86265ef-a35f-4c8b-9e28-59480c6d3657","Type":"ContainerStarted","Data":"f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a"}
Oct 30 22:01:54.222573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:54.222069    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" event={"ID":"b86265ef-a35f-4c8b-9e28-59480c6d3657","Type":"ContainerStarted","Data":"43e8cbafd1aa17c9c2950e451c57ed40b5f0a4142bfa8c52cfbc475a13528259"}
Oct 30 22:01:54.373229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:54.373164    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podStartSLOduration=2.373148763 podStartE2EDuration="2.373148763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:54.345799119 +0000 UTC m=+725.869941896" watchObservedRunningTime="2026-10-30 22:01:54.373148763 +0000 UTC m=+725.897291527"
Oct 30 22:01:54.454561 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:54.454526    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:01:54.885951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:54.885917    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:54.993293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:54.993245    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:01:59.997915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:01:59.997880    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:02.036956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:02.036923    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:02.993830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:02.993794    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:02:03.007697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:03.007668    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:02:03.310754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:03.310634    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:02:05.151586 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:05.151545    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:10.229081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:10.229049    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:11.995483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:11.995442    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:02:15.302658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:15.302620    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:17.026253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.026221    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"]
Oct 30 22:02:17.030376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.030359    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.068564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.068532    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tls-certs\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.068723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.068573    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kb7q\" (UniqueName: \"kubernetes.io/projected/ff7a540c-99bf-420d-a7eb-ef1664d01bce-kube-api-access-2kb7q\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.068723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.068709    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tmp-dir\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.068847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.068752    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-model-cache\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.068975 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.068952    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-home\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.069039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.069011    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-dshm\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.126266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.126238    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"openshift-service-ca.crt\""
Oct 30 22:02:17.126426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.126273    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"kube-root-ca.crt\""
Oct 30 22:02:17.141507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.141481    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"default-dockercfg-tjwnv\""
Oct 30 22:02:17.142008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.141995    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"simulator-w4-bce513-kserve-self-signed-certs\""
Oct 30 22:02:17.144420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.144402    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"]
Oct 30 22:02:17.169961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.169928    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tmp-dir\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.170311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.170286    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-model-cache\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.170792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.170666    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tmp-dir\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.170919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.170807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-home\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.170919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.170680    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-model-cache\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.170919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.170871    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-dshm\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.170919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.170899    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tls-certs\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.171271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.170932    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kb7q\" (UniqueName: \"kubernetes.io/projected/ff7a540c-99bf-420d-a7eb-ef1664d01bce-kube-api-access-2kb7q\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.171271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.171130    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-home\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.174829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.174794    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-dshm\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.175716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.175686    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tls-certs\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.194566 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.194531    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:17.240447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.240421    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kb7q\" (UniqueName: \"kubernetes.io/projected/ff7a540c-99bf-420d-a7eb-ef1664d01bce-kube-api-access-2kb7q\") pod \"simulator-w4-bce513-kserve-695578ddd8-mg4tz\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") " pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.343198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.343171    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:17.421413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.421379    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:17.735852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:17.735814    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"]
Oct 30 22:02:17.736677 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:02:17.736646    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff7a540c_99bf_420d_a7eb_ef1664d01bce.slice/crio-1051d13b79f78819295503ed7446287c476775701e1dbc2b023abb8e3678f6a0 WatchSource:0}: Error finding container 1051d13b79f78819295503ed7446287c476775701e1dbc2b023abb8e3678f6a0: Status 404 returned error can't find the container with id 1051d13b79f78819295503ed7446287c476775701e1dbc2b023abb8e3678f6a0
Oct 30 22:02:18.402522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:18.402487    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" event={"ID":"ff7a540c-99bf-420d-a7eb-ef1664d01bce","Type":"ContainerStarted","Data":"0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2"}
Oct 30 22:02:18.402522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:18.402530    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" event={"ID":"ff7a540c-99bf-420d-a7eb-ef1664d01bce","Type":"ContainerStarted","Data":"1051d13b79f78819295503ed7446287c476775701e1dbc2b023abb8e3678f6a0"}
Oct 30 22:02:18.524176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:18.524122    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podStartSLOduration=2.524108824 podStartE2EDuration="2.524108824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:18.523093517 +0000 UTC m=+750.047236293" watchObservedRunningTime="2026-10-30 22:02:18.524108824 +0000 UTC m=+750.048251588"
Oct 30 22:02:19.343773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:19.343741    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:20.299935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:20.299903    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:20.541835 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:02:20.541789    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:02:25.433928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:25.433897    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:27.343640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:27.343602    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:27.357782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:27.357722    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:27.476313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:27.476283    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:02:28.636225 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:28.636183    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:30.430305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:30.430261    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:35.505045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:35.505015    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:36.908801 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:36.908770    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:02:40.601637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:40.601600    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:42.992616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:42.992585    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:45.265847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:45.265813    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:45.621177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:45.621148    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:50.544945 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:02:50.544902    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:02:50.647451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:50.647389    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:55.829988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:02:55.829953    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:00.916437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:00.916396    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:06.047720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:06.047670    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:07.419370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:07.419330    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:03:11.121404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:11.121364    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:11.930901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:11.930870    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:16.106482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:16.106443    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:20.543827 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:03:20.543780    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:03:21.195616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:21.195585    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:25.759384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.759345    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"]
Oct 30 22:03:25.763799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.763768    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.778802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.778770    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-dshm\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.778961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.778810    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452a0cbb-a40b-4d70-972e-59ad3380495d-tls-certs\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.778961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.778840    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-home\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.778961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.778907    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfvp2\" (UniqueName: \"kubernetes.io/projected/452a0cbb-a40b-4d70-972e-59ad3380495d-kube-api-access-kfvp2\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.778961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.778935    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-model-cache\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.779130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.778966    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-tmp-dir\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.873906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.873874    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2182a5\"/\"premium-w6-2182a5-kserve-self-signed-certs\""
Oct 30 22:03:25.877237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.877201    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"]
Oct 30 22:03:25.879702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.879679    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-tmp-dir\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.879866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.879848    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-dshm\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.879908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.879883    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452a0cbb-a40b-4d70-972e-59ad3380495d-tls-certs\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.879958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.879913    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-home\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.880113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.880093    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfvp2\" (UniqueName: \"kubernetes.io/projected/452a0cbb-a40b-4d70-972e-59ad3380495d-kube-api-access-kfvp2\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.880197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.880140    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-model-cache\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.880283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.880141    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-tmp-dir\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.880338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.880289    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-home\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.880495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.880471    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-model-cache\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.882374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.882355    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-dshm\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.882598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.882580    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452a0cbb-a40b-4d70-972e-59ad3380495d-tls-certs\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.944080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.944043    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfvp2\" (UniqueName: \"kubernetes.io/projected/452a0cbb-a40b-4d70-972e-59ad3380495d-kube-api-access-kfvp2\") pod \"premium-w6-2182a5-kserve-84bc64fb59-pw4c4\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") " pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:25.988868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:25.988817    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:26.076881 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:26.076784    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:26.346098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:26.346064    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:26.493342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:26.493313    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"]
Oct 30 22:03:26.494182 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:03:26.494149    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod452a0cbb_a40b_4d70_972e_59ad3380495d.slice/crio-428259637b37bb987340a122fb6c78d2c954f3cd460cdbf8af0852853512c83c WatchSource:0}: Error finding container 428259637b37bb987340a122fb6c78d2c954f3cd460cdbf8af0852853512c83c: Status 404 returned error can't find the container with id 428259637b37bb987340a122fb6c78d2c954f3cd460cdbf8af0852853512c83c
Oct 30 22:03:26.881979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:26.881947    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" event={"ID":"452a0cbb-a40b-4d70-972e-59ad3380495d","Type":"ContainerStarted","Data":"cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6"}
Oct 30 22:03:26.882510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:26.881994    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" event={"ID":"452a0cbb-a40b-4d70-972e-59ad3380495d","Type":"ContainerStarted","Data":"428259637b37bb987340a122fb6c78d2c954f3cd460cdbf8af0852853512c83c"}
Oct 30 22:03:27.005379 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:27.005323    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podStartSLOduration=2.005309425 podStartE2EDuration="2.005309425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:27.003185165 +0000 UTC m=+818.527327949" watchObservedRunningTime="2026-10-30 22:03:27.005309425 +0000 UTC m=+818.529452188"
Oct 30 22:03:27.077595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:27.077564    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:31.418380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:31.418340    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:36.077363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.077332    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:36.090609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.090584    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:36.334013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.333920    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:36.558616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.558576    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:36.723392 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.723357    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"]
Oct 30 22:03:36.727470 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.727451    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.783702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.783670    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:36.783900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.783732    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:36.786740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.786711    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"]
Oct 30 22:03:36.811433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.811405    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.811619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.811474    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzgt\" (UniqueName: \"kubernetes.io/projected/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-kube-api-access-2pzgt\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.811619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.811582    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.833546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.833519    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:36.833546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.833534    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:03:36.912964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.912931    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.913146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.913025    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzgt\" (UniqueName: \"kubernetes.io/projected/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-kube-api-access-2pzgt\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.913146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.913056    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.913806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.913782    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.915518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.915501    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:36.963773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.963745    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:03:36.977226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:36.977146    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzgt\" (UniqueName: \"kubernetes.io/projected/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-kube-api-access-2pzgt\") pod \"maas-api-revoke-sub-55e454ce-cxjv4\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:37.044400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:37.044374    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:37.427529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:37.427500    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"]
Oct 30 22:03:37.428908 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:03:37.428874    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81eadeb_a6f1_4fd1_9db9_e72a9bbf165f.slice/crio-8bb2ea2074ab9a7d7785f1dce14164a4bdf183a3b6743a5143ad581a6e1371fc WatchSource:0}: Error finding container 8bb2ea2074ab9a7d7785f1dce14164a4bdf183a3b6743a5143ad581a6e1371fc: Status 404 returned error can't find the container with id 8bb2ea2074ab9a7d7785f1dce14164a4bdf183a3b6743a5143ad581a6e1371fc
Oct 30 22:03:37.960984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:37.960941    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4" event={"ID":"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f","Type":"ContainerStarted","Data":"1080ebfb0f7e200299a0eb953129132c62117c1f0b7db8fcb5972ff7722e9cec"}
Oct 30 22:03:37.961196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:37.961003    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4" event={"ID":"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f","Type":"ContainerStarted","Data":"8bb2ea2074ab9a7d7785f1dce14164a4bdf183a3b6743a5143ad581a6e1371fc"}
Oct 30 22:03:38.084875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:38.084770    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4" podStartSLOduration=2.084755878 podStartE2EDuration="2.084755878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:38.081923689 +0000 UTC m=+829.606066453" watchObservedRunningTime="2026-10-30 22:03:38.084755878 +0000 UTC m=+829.608898642"
Oct 30 22:03:38.969431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:38.969394    5031 generic.go:356] "Generic (PLEG): container finished" podID="e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" containerID="1080ebfb0f7e200299a0eb953129132c62117c1f0b7db8fcb5972ff7722e9cec" exitCode=0
Oct 30 22:03:38.969812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:38.969481    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4" event={"ID":"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f","Type":"ContainerDied","Data":"1080ebfb0f7e200299a0eb953129132c62117c1f0b7db8fcb5972ff7722e9cec"}
Oct 30 22:03:40.129027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.128981    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:40.252990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.252895    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-xks-ca\") pod \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") "
Oct 30 22:03:40.252990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.252972    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-kube-api-access-2pzgt\" (UniqueName: \"kubernetes.io/projected/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-kube-api-access-2pzgt\") pod \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") "
Oct 30 22:03:40.253272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.253041    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-openshift-ca\") pod \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\" (UID: \"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f\") "
Oct 30 22:03:40.254117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.253812    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-openshift-ca" pod "e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" (UID: "e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:40.256739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.256712    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-kube-api-access-2pzgt" pod "e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" (UID: "e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f"). InnerVolumeSpecName "kube-api-access-2pzgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:40.257639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.257615    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-xks-ca" pod "e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" (UID: "e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:40.355180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.355139    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.355180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.355178    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzgt\" (UniqueName: \"kubernetes.io/projected/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-kube-api-access-2pzgt\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.355180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.355191    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:40.983770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.983689    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"
Oct 30 22:03:40.983931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.983689    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4" event={"ID":"e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f","Type":"ContainerDied","Data":"8bb2ea2074ab9a7d7785f1dce14164a4bdf183a3b6743a5143ad581a6e1371fc"}
Oct 30 22:03:40.983931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:40.983833    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb2ea2074ab9a7d7785f1dce14164a4bdf183a3b6743a5143ad581a6e1371fc"
Oct 30 22:03:41.413694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:41.413616    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:42.548234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:42.547708    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:03:43.334502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.334452    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:43.697180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.697135    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"]
Oct 30 22:03:43.698047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.698014    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" containerName="revoke-keys"
Oct 30 22:03:43.698047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.698041    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" containerName="revoke-keys"
Oct 30 22:03:43.698262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.698158    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" containerName="revoke-keys"
Oct 30 22:03:43.701698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.701682    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.765033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.764977    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"]
Oct 30 22:03:43.795649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.795608    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-tmp-dir\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.796013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.795991    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-model-cache\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.796258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.796181    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqghs\" (UniqueName: \"kubernetes.io/projected/b27147da-5731-4444-92d8-58f799907b2b-kube-api-access-cqghs\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.796351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.796274    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-dshm\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.796410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.796398    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-home\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.796477 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.796457    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27147da-5731-4444-92d8-58f799907b2b-tls-certs\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.809781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.809756    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"premium-w4-bce513-kserve-self-signed-certs\""
Oct 30 22:03:43.876337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.876302    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:43.898019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.897988    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-model-cache\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898029    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqghs\" (UniqueName: \"kubernetes.io/projected/b27147da-5731-4444-92d8-58f799907b2b-kube-api-access-cqghs\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898088    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-dshm\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898192    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-home\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898263    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27147da-5731-4444-92d8-58f799907b2b-tls-certs\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898315    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-tmp-dir\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898629    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-model-cache\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898677    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-tmp-dir\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.898831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.898771    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-home\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.900593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.900573    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-dshm\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.901643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.901614    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27147da-5731-4444-92d8-58f799907b2b-tls-certs\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:43.968269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:43.968154    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqghs\" (UniqueName: \"kubernetes.io/projected/b27147da-5731-4444-92d8-58f799907b2b-kube-api-access-cqghs\") pod \"premium-w4-bce513-kserve-798dd849b5-54kfv\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") " pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:44.014332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:44.014242    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:44.465379 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:03:44.465344    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb27147da_5731_4444_92d8_58f799907b2b.slice/crio-fb0742930c9d8d6f2c030ade4b9313d24148e0041acda8a50e32142dabb175ee WatchSource:0}: Error finding container fb0742930c9d8d6f2c030ade4b9313d24148e0041acda8a50e32142dabb175ee: Status 404 returned error can't find the container with id fb0742930c9d8d6f2c030ade4b9313d24148e0041acda8a50e32142dabb175ee
Oct 30 22:03:44.469737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:44.469713    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"]
Oct 30 22:03:45.015819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:45.015777    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" event={"ID":"b27147da-5731-4444-92d8-58f799907b2b","Type":"ContainerStarted","Data":"fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81"}
Oct 30 22:03:45.016278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:45.015829    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" event={"ID":"b27147da-5731-4444-92d8-58f799907b2b","Type":"ContainerStarted","Data":"fb0742930c9d8d6f2c030ade4b9313d24148e0041acda8a50e32142dabb175ee"}
Oct 30 22:03:45.138530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:45.138469    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podStartSLOduration=2.138454986 podStartE2EDuration="2.138454986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:45.136658546 +0000 UTC m=+836.660801330" watchObservedRunningTime="2026-10-30 22:03:45.138454986 +0000 UTC m=+836.662597750"
Oct 30 22:03:46.015277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:46.015233    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:46.587992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:46.587960    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:50.541581 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:03:50.541541    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:03:51.715643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:51.715601    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:52.324826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.324792    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"]
Oct 30 22:03:52.329037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.329018    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.383781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.383755    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:52.434530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.434506    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:52.439871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.439843    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"]
Oct 30 22:03:52.447580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.447560    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:52.484678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.484653    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:03:52.505382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.505356    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.505568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.505546    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njr5c\" (UniqueName: \"kubernetes.io/projected/c2659a03-b3c3-4ef5-8735-ff7b277d811f-kube-api-access-njr5c\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.505722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.505700    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.606716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.606687    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.606879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.606765    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.606935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.606918    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njr5c\" (UniqueName: \"kubernetes.io/projected/c2659a03-b3c3-4ef5-8735-ff7b277d811f-kube-api-access-njr5c\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.607393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.607367    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.609336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.609315    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.675608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.675511    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njr5c\" (UniqueName: \"kubernetes.io/projected/c2659a03-b3c3-4ef5-8735-ff7b277d811f-kube-api-access-njr5c\") pod \"maas-api-revoke-sub-546bcf35-m48x7\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:52.969282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:52.968192    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:53.352999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:53.352969    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"]
Oct 30 22:03:53.354684 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:03:53.354650    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2659a03_b3c3_4ef5_8735_ff7b277d811f.slice/crio-9b8680832ce6506f55d78788cc3939c43947eb893d353f122995a4f43dc751b5 WatchSource:0}: Error finding container 9b8680832ce6506f55d78788cc3939c43947eb893d353f122995a4f43dc751b5: Status 404 returned error can't find the container with id 9b8680832ce6506f55d78788cc3939c43947eb893d353f122995a4f43dc751b5
Oct 30 22:03:54.014634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:54.014540    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:54.027665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:54.027637    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:54.086861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:54.086827    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7" event={"ID":"c2659a03-b3c3-4ef5-8735-ff7b277d811f","Type":"ContainerStarted","Data":"2e80214dec4825ae13c8414e05667751a145b865c18f81265f796fd82e30ac3b"}
Oct 30 22:03:54.087020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:54.086884    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7" event={"ID":"c2659a03-b3c3-4ef5-8735-ff7b277d811f","Type":"ContainerStarted","Data":"9b8680832ce6506f55d78788cc3939c43947eb893d353f122995a4f43dc751b5"}
Oct 30 22:03:54.098312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:54.098287    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:03:54.269742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:54.269649    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7" podStartSLOduration=2.269629509 podStartE2EDuration="2.269629509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:54.269178233 +0000 UTC m=+845.793321015" watchObservedRunningTime="2026-10-30 22:03:54.269629509 +0000 UTC m=+845.793772277"
Oct 30 22:03:55.096551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:55.096512    5031 generic.go:356] "Generic (PLEG): container finished" podID="c2659a03-b3c3-4ef5-8735-ff7b277d811f" containerID="2e80214dec4825ae13c8414e05667751a145b865c18f81265f796fd82e30ac3b" exitCode=0
Oct 30 22:03:55.096995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:55.096608    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7" event={"ID":"c2659a03-b3c3-4ef5-8735-ff7b277d811f","Type":"ContainerDied","Data":"2e80214dec4825ae13c8414e05667751a145b865c18f81265f796fd82e30ac3b"}
Oct 30 22:03:56.249412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.249386    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:03:56.351424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.351392    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-xks-ca\") pod \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") "
Oct 30 22:03:56.351599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.351446    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2659a03-b3c3-4ef5-8735-ff7b277d811f-kube-api-access-njr5c\" (UniqueName: \"kubernetes.io/projected/c2659a03-b3c3-4ef5-8735-ff7b277d811f-kube-api-access-njr5c\") pod \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") "
Oct 30 22:03:56.351641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.351618    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-openshift-ca\") pod \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\" (UID: \"c2659a03-b3c3-4ef5-8735-ff7b277d811f\") "
Oct 30 22:03:56.351949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.351924    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-openshift-ca" pod "c2659a03-b3c3-4ef5-8735-ff7b277d811f" (UID: "c2659a03-b3c3-4ef5-8735-ff7b277d811f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:56.352134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.352117    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:56.353753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.353734    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2659a03-b3c3-4ef5-8735-ff7b277d811f-kube-api-access-njr5c" pod "c2659a03-b3c3-4ef5-8735-ff7b277d811f" (UID: "c2659a03-b3c3-4ef5-8735-ff7b277d811f"). InnerVolumeSpecName "kube-api-access-njr5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:56.353931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.353918    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-xks-ca" pod "c2659a03-b3c3-4ef5-8735-ff7b277d811f" (UID: "c2659a03-b3c3-4ef5-8735-ff7b277d811f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:56.453682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.453639    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2659a03-b3c3-4ef5-8735-ff7b277d811f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:56.453682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.453673    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njr5c\" (UniqueName: \"kubernetes.io/projected/c2659a03-b3c3-4ef5-8735-ff7b277d811f-kube-api-access-njr5c\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:56.717160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:56.717127    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:57.111805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:57.111764    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7" event={"ID":"c2659a03-b3c3-4ef5-8735-ff7b277d811f","Type":"ContainerDied","Data":"9b8680832ce6506f55d78788cc3939c43947eb893d353f122995a4f43dc751b5"}
Oct 30 22:03:57.112002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:57.111826    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8680832ce6506f55d78788cc3939c43947eb893d353f122995a4f43dc751b5"
Oct 30 22:03:57.112002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:03:57.111775    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"
Oct 30 22:04:00.895275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:00.895243    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"]
Oct 30 22:04:00.895866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:00.895851    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c2659a03-b3c3-4ef5-8735-ff7b277d811f" containerName="revoke-keys"
Oct 30 22:04:00.895913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:00.895868    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2659a03-b3c3-4ef5-8735-ff7b277d811f" containerName="revoke-keys"
Oct 30 22:04:00.895995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:00.895985    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2659a03-b3c3-4ef5-8735-ff7b277d811f" containerName="revoke-keys"
Oct 30 22:04:00.899830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:00.899813    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:00.959976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:00.959940    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"]
Oct 30 22:04:01.002547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.002509    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.002716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.002630    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.002716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.002677    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddcb\" (UniqueName: \"kubernetes.io/projected/7426b215-5e4f-4391-bcbb-a5fbd38422ce-kube-api-access-jddcb\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.005573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.005547    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:01.005573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.005547    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:01.005751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.005624    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:01.005751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.005653    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:04:01.104343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.104309    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.104501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.104357    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddcb\" (UniqueName: \"kubernetes.io/projected/7426b215-5e4f-4391-bcbb-a5fbd38422ce-kube-api-access-jddcb\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.104501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.104477    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.105165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.105032    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.106986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.106962    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.170912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.170842    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddcb\" (UniqueName: \"kubernetes.io/projected/7426b215-5e4f-4391-bcbb-a5fbd38422ce-kube-api-access-jddcb\") pod \"maas-api-revoke-sub-81cba0bf-tl49c\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.215542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.215509    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:01.601358 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:04:01.601325    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7426b215_5e4f_4391_bcbb_a5fbd38422ce.slice/crio-cb9fce957ac693d7ebc5fedadd4dd8aa7e0e06eba0511fcab8caea0a90b4b7f9 WatchSource:0}: Error finding container cb9fce957ac693d7ebc5fedadd4dd8aa7e0e06eba0511fcab8caea0a90b4b7f9: Status 404 returned error can't find the container with id cb9fce957ac693d7ebc5fedadd4dd8aa7e0e06eba0511fcab8caea0a90b4b7f9
Oct 30 22:04:01.601510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.601423    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"]
Oct 30 22:04:01.715418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:01.715375    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:02.156618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:02.156581    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c" event={"ID":"7426b215-5e4f-4391-bcbb-a5fbd38422ce","Type":"ContainerStarted","Data":"fed45606bf5e1d9ca5d311c925395b2380911b8782de9f32a2ffb5bb70ca25bd"}
Oct 30 22:04:02.157127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:02.156646    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c" event={"ID":"7426b215-5e4f-4391-bcbb-a5fbd38422ce","Type":"ContainerStarted","Data":"cb9fce957ac693d7ebc5fedadd4dd8aa7e0e06eba0511fcab8caea0a90b4b7f9"}
Oct 30 22:04:02.277759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:02.277648    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c" podStartSLOduration=2.277633714 podStartE2EDuration="2.277633714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:02.275591575 +0000 UTC m=+853.799734339" watchObservedRunningTime="2026-10-30 22:04:02.277633714 +0000 UTC m=+853.801776509"
Oct 30 22:04:03.165819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:03.165776    5031 generic.go:356] "Generic (PLEG): container finished" podID="7426b215-5e4f-4391-bcbb-a5fbd38422ce" containerID="fed45606bf5e1d9ca5d311c925395b2380911b8782de9f32a2ffb5bb70ca25bd" exitCode=0
Oct 30 22:04:03.166320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:03.165861    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c" event={"ID":"7426b215-5e4f-4391-bcbb-a5fbd38422ce","Type":"ContainerDied","Data":"fed45606bf5e1d9ca5d311c925395b2380911b8782de9f32a2ffb5bb70ca25bd"}
Oct 30 22:04:04.434961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.434887    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:04.459621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.459580    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7426b215-5e4f-4391-bcbb-a5fbd38422ce-kube-api-access-jddcb\" (UniqueName: \"kubernetes.io/projected/7426b215-5e4f-4391-bcbb-a5fbd38422ce-kube-api-access-jddcb\") pod \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") "
Oct 30 22:04:04.459999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.459972    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-xks-ca\") pod \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") "
Oct 30 22:04:04.460326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.460305    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-openshift-ca\") pod \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\" (UID: \"7426b215-5e4f-4391-bcbb-a5fbd38422ce\") "
Oct 30 22:04:04.493254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.485933    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7426b215-5e4f-4391-bcbb-a5fbd38422ce-kube-api-access-jddcb" pod "7426b215-5e4f-4391-bcbb-a5fbd38422ce" (UID: "7426b215-5e4f-4391-bcbb-a5fbd38422ce"). InnerVolumeSpecName "kube-api-access-jddcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:04.493254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.492755    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-openshift-ca" pod "7426b215-5e4f-4391-bcbb-a5fbd38422ce" (UID: "7426b215-5e4f-4391-bcbb-a5fbd38422ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:04.494766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.494730    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-xks-ca" pod "7426b215-5e4f-4391-bcbb-a5fbd38422ce" (UID: "7426b215-5e4f-4391-bcbb-a5fbd38422ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:04.599575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.599447    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:04.599575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.599485    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddcb\" (UniqueName: \"kubernetes.io/projected/7426b215-5e4f-4391-bcbb-a5fbd38422ce-kube-api-access-jddcb\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:04.599575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:04.599502    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7426b215-5e4f-4391-bcbb-a5fbd38422ce-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:05.185783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:05.185751    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c" event={"ID":"7426b215-5e4f-4391-bcbb-a5fbd38422ce","Type":"ContainerDied","Data":"cb9fce957ac693d7ebc5fedadd4dd8aa7e0e06eba0511fcab8caea0a90b4b7f9"}
Oct 30 22:04:05.185783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:05.185781    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"
Oct 30 22:04:05.185783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:05.185800    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb9fce957ac693d7ebc5fedadd4dd8aa7e0e06eba0511fcab8caea0a90b4b7f9"
Oct 30 22:04:07.040730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:07.040597    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:08.281419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.281378    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"]
Oct 30 22:04:08.282286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.282242    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7426b215-5e4f-4391-bcbb-a5fbd38422ce" containerName="revoke-keys"
Oct 30 22:04:08.282286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.282266    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7426b215-5e4f-4391-bcbb-a5fbd38422ce" containerName="revoke-keys"
Oct 30 22:04:08.282497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.282484    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7426b215-5e4f-4391-bcbb-a5fbd38422ce" containerName="revoke-keys"
Oct 30 22:04:08.286859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.286837    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.344651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.344606    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"]
Oct 30 22:04:08.357307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.357270    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8xp\" (UniqueName: \"kubernetes.io/projected/f34fba32-6865-4772-86b8-9a706361fd27-kube-api-access-2d8xp\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.357528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.357501    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.357635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.357615    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.390312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.390282    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:08.390531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.390508    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:04:08.390587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.390556    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:08.392910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.392885    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:08.458994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.458963    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8xp\" (UniqueName: \"kubernetes.io/projected/f34fba32-6865-4772-86b8-9a706361fd27-kube-api-access-2d8xp\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.459168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.459056    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.459168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.459100    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.459726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.459701    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.461659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.461632    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.522870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.522833    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8xp\" (UniqueName: \"kubernetes.io/projected/f34fba32-6865-4772-86b8-9a706361fd27-kube-api-access-2d8xp\") pod \"maas-api-revoke-sub-cb37fc14-mp5cl\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.571175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.571098    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:08.603262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.603231    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:08.996701 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:04:08.996670    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34fba32_6865_4772_86b8_9a706361fd27.slice/crio-491249661b5e8e3f34d94258a9db329b69ce99970fc3515b340b79ee5efc20e3 WatchSource:0}: Error finding container 491249661b5e8e3f34d94258a9db329b69ce99970fc3515b340b79ee5efc20e3: Status 404 returned error can't find the container with id 491249661b5e8e3f34d94258a9db329b69ce99970fc3515b340b79ee5efc20e3
Oct 30 22:04:08.996817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:08.996779    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"]
Oct 30 22:04:09.216137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:09.216104    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl" event={"ID":"f34fba32-6865-4772-86b8-9a706361fd27","Type":"ContainerStarted","Data":"491249661b5e8e3f34d94258a9db329b69ce99970fc3515b340b79ee5efc20e3"}
Oct 30 22:04:10.225843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:10.225744    5031 generic.go:356] "Generic (PLEG): container finished" podID="f34fba32-6865-4772-86b8-9a706361fd27" containerID="5e618d910c16bc9a7d9904f0e3190851956d21e217fc0a30beffa74336a4f4ec" exitCode=0
Oct 30 22:04:10.225843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:10.225800    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl" event={"ID":"f34fba32-6865-4772-86b8-9a706361fd27","Type":"ContainerDied","Data":"5e618d910c16bc9a7d9904f0e3190851956d21e217fc0a30beffa74336a4f4ec"}
Oct 30 22:04:11.402845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.402817    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:11.500467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.500387    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34fba32-6865-4772-86b8-9a706361fd27-kube-api-access-2d8xp\" (UniqueName: \"kubernetes.io/projected/f34fba32-6865-4772-86b8-9a706361fd27-kube-api-access-2d8xp\") pod \"f34fba32-6865-4772-86b8-9a706361fd27\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") "
Oct 30 22:04:11.500467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.500425    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-openshift-ca\") pod \"f34fba32-6865-4772-86b8-9a706361fd27\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") "
Oct 30 22:04:11.500680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.500560    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-xks-ca\") pod \"f34fba32-6865-4772-86b8-9a706361fd27\" (UID: \"f34fba32-6865-4772-86b8-9a706361fd27\") "
Oct 30 22:04:11.500798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.500777    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-openshift-ca" pod "f34fba32-6865-4772-86b8-9a706361fd27" (UID: "f34fba32-6865-4772-86b8-9a706361fd27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:11.501055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.501041    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:11.502602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.502582    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34fba32-6865-4772-86b8-9a706361fd27-kube-api-access-2d8xp" pod "f34fba32-6865-4772-86b8-9a706361fd27" (UID: "f34fba32-6865-4772-86b8-9a706361fd27"). InnerVolumeSpecName "kube-api-access-2d8xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:11.502697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.502662    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-xks-ca" pod "f34fba32-6865-4772-86b8-9a706361fd27" (UID: "f34fba32-6865-4772-86b8-9a706361fd27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:11.602609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.602562    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8xp\" (UniqueName: \"kubernetes.io/projected/f34fba32-6865-4772-86b8-9a706361fd27-kube-api-access-2d8xp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:11.602609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.602603    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34fba32-6865-4772-86b8-9a706361fd27-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:11.758830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.758746    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"]
Oct 30 22:04:11.759438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.759417    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f34fba32-6865-4772-86b8-9a706361fd27" containerName="revoke-keys"
Oct 30 22:04:11.759558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.759437    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34fba32-6865-4772-86b8-9a706361fd27" containerName="revoke-keys"
Oct 30 22:04:11.759621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.759610    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34fba32-6865-4772-86b8-9a706361fd27" containerName="revoke-keys"
Oct 30 22:04:11.763047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.763027    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:11.822121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.822090    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"]
Oct 30 22:04:11.906595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.906565    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:11.906848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.906714    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rcz\" (UniqueName: \"kubernetes.io/projected/bee75358-a467-4461-98ee-dfb8a1911764-kube-api-access-b4rcz\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:11.906848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:11.906821    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.008340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.008299    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.009601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.009513    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.009766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.009647    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rcz\" (UniqueName: \"kubernetes.io/projected/bee75358-a467-4461-98ee-dfb8a1911764-kube-api-access-b4rcz\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.010118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.009397    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.013090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.013058    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.040368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.040335    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:12.073613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.073587    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rcz\" (UniqueName: \"kubernetes.io/projected/bee75358-a467-4461-98ee-dfb8a1911764-kube-api-access-b4rcz\") pod \"maas-api-revoke-sub-1cc06c38-tckjn\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.078423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.078403    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:12.242431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.242403    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"
Oct 30 22:04:12.242602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.242402    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl" event={"ID":"f34fba32-6865-4772-86b8-9a706361fd27","Type":"ContainerDied","Data":"491249661b5e8e3f34d94258a9db329b69ce99970fc3515b340b79ee5efc20e3"}
Oct 30 22:04:12.242602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.242531    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491249661b5e8e3f34d94258a9db329b69ce99970fc3515b340b79ee5efc20e3"
Oct 30 22:04:12.463455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:12.463422    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"]
Oct 30 22:04:12.464319 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:04:12.464293    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee75358_a467_4461_98ee_dfb8a1911764.slice/crio-6fcce633630e9958fd0e5237ffcd07a337431f43c84fae6807ad19fd7a495569 WatchSource:0}: Error finding container 6fcce633630e9958fd0e5237ffcd07a337431f43c84fae6807ad19fd7a495569: Status 404 returned error can't find the container with id 6fcce633630e9958fd0e5237ffcd07a337431f43c84fae6807ad19fd7a495569
Oct 30 22:04:13.251590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:13.251554    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn" event={"ID":"bee75358-a467-4461-98ee-dfb8a1911764","Type":"ContainerStarted","Data":"0a4d61185a7ee3bc29f3b14e07c5b37095cf4766df914b7f3e9ab11716649a73"}
Oct 30 22:04:13.251789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:13.251612    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn" event={"ID":"bee75358-a467-4461-98ee-dfb8a1911764","Type":"ContainerStarted","Data":"6fcce633630e9958fd0e5237ffcd07a337431f43c84fae6807ad19fd7a495569"}
Oct 30 22:04:13.374480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:13.374424    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn" podStartSLOduration=2.374410227 podStartE2EDuration="2.374410227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:13.371996546 +0000 UTC m=+864.896139327" watchObservedRunningTime="2026-10-30 22:04:13.374410227 +0000 UTC m=+864.898552992"
Oct 30 22:04:14.263280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:14.263237    5031 generic.go:356] "Generic (PLEG): container finished" podID="bee75358-a467-4461-98ee-dfb8a1911764" containerID="0a4d61185a7ee3bc29f3b14e07c5b37095cf4766df914b7f3e9ab11716649a73" exitCode=0
Oct 30 22:04:14.264162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:14.263380    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn" event={"ID":"bee75358-a467-4461-98ee-dfb8a1911764","Type":"ContainerDied","Data":"0a4d61185a7ee3bc29f3b14e07c5b37095cf4766df914b7f3e9ab11716649a73"}
Oct 30 22:04:15.429220 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.429171    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:15.549871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.549774    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-openshift-ca\") pod \"bee75358-a467-4461-98ee-dfb8a1911764\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") "
Oct 30 22:04:15.549871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.549819    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee75358-a467-4461-98ee-dfb8a1911764-kube-api-access-b4rcz\" (UniqueName: \"kubernetes.io/projected/bee75358-a467-4461-98ee-dfb8a1911764-kube-api-access-b4rcz\") pod \"bee75358-a467-4461-98ee-dfb8a1911764\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") "
Oct 30 22:04:15.550125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.549976    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-xks-ca\") pod \"bee75358-a467-4461-98ee-dfb8a1911764\" (UID: \"bee75358-a467-4461-98ee-dfb8a1911764\") "
Oct 30 22:04:15.550275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.550244    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-openshift-ca" pod "bee75358-a467-4461-98ee-dfb8a1911764" (UID: "bee75358-a467-4461-98ee-dfb8a1911764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:15.550433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.550415    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:15.552190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.552164    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-xks-ca" pod "bee75358-a467-4461-98ee-dfb8a1911764" (UID: "bee75358-a467-4461-98ee-dfb8a1911764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:15.552261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.552223    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee75358-a467-4461-98ee-dfb8a1911764-kube-api-access-b4rcz" pod "bee75358-a467-4461-98ee-dfb8a1911764" (UID: "bee75358-a467-4461-98ee-dfb8a1911764"). InnerVolumeSpecName "kube-api-access-b4rcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:15.651648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.651606    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rcz\" (UniqueName: \"kubernetes.io/projected/bee75358-a467-4461-98ee-dfb8a1911764-kube-api-access-b4rcz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:15.651648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:15.651646    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee75358-a467-4461-98ee-dfb8a1911764-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:16.278441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:16.278412    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"
Oct 30 22:04:16.278634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:16.278461    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn" event={"ID":"bee75358-a467-4461-98ee-dfb8a1911764","Type":"ContainerDied","Data":"6fcce633630e9958fd0e5237ffcd07a337431f43c84fae6807ad19fd7a495569"}
Oct 30 22:04:16.278634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:16.278512    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fcce633630e9958fd0e5237ffcd07a337431f43c84fae6807ad19fd7a495569"
Oct 30 22:04:17.149978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:17.149942    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:20.541106 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:04:20.541065    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:04:22.282152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:22.282107    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:23.006559 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:23.006514    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:04:27.488957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:27.488921    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:32.601771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:32.601738    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:35.199338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.199304    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"]
Oct 30 22:04:35.199903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.199889    5031 cpu_manager.go:414] "containerMap: removing container" podUID="bee75358-a467-4461-98ee-dfb8a1911764" containerName="revoke-keys"
Oct 30 22:04:35.199962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.199905    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee75358-a467-4461-98ee-dfb8a1911764" containerName="revoke-keys"
Oct 30 22:04:35.200038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.200028    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee75358-a467-4461-98ee-dfb8a1911764" containerName="revoke-keys"
Oct 30 22:04:35.203522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.203506    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.259771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.259744    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:35.259771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.259765    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:35.295474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.295442    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.295830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.295800    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.296011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.295973    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7mg\" (UniqueName: \"kubernetes.io/projected/d4994728-c514-4fbc-a183-61eb70d1648c-kube-api-access-st7mg\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.310202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.310182    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:35.310344 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.310319    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:04:35.314045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.314011    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"]
Oct 30 22:04:35.396852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.396825    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.397031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.396919    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7mg\" (UniqueName: \"kubernetes.io/projected/d4994728-c514-4fbc-a183-61eb70d1648c-kube-api-access-st7mg\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.397031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.396969    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.397568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.397543    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.399576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.399554    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.462447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.462374    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7mg\" (UniqueName: \"kubernetes.io/projected/d4994728-c514-4fbc-a183-61eb70d1648c-kube-api-access-st7mg\") pod \"maas-api-revoke-sub-15a989cb-zt9tx\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.517180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.517148    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:35.902252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:35.902228    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"]
Oct 30 22:04:35.902865 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:04:35.902838    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4994728_c514_4fbc_a183_61eb70d1648c.slice/crio-7a9f39b246d84c7abd35d569ad649ec4f192f7257b4dc6fc63590afe02ea5ed7 WatchSource:0}: Error finding container 7a9f39b246d84c7abd35d569ad649ec4f192f7257b4dc6fc63590afe02ea5ed7: Status 404 returned error can't find the container with id 7a9f39b246d84c7abd35d569ad649ec4f192f7257b4dc6fc63590afe02ea5ed7
Oct 30 22:04:36.459417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:36.459378    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx" event={"ID":"d4994728-c514-4fbc-a183-61eb70d1648c","Type":"ContainerStarted","Data":"932c63de613e1e15f1f730f51c0b305e279b5a95ef2d32e72ae7cc6e8a937b56"}
Oct 30 22:04:36.459875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:36.459435    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx" event={"ID":"d4994728-c514-4fbc-a183-61eb70d1648c","Type":"ContainerStarted","Data":"7a9f39b246d84c7abd35d569ad649ec4f192f7257b4dc6fc63590afe02ea5ed7"}
Oct 30 22:04:36.578980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:36.578884    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx" podStartSLOduration=1.578870415 podStartE2EDuration="1.578870415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:36.578632656 +0000 UTC m=+888.102775471" watchObservedRunningTime="2026-10-30 22:04:36.578870415 +0000 UTC m=+888.103013178"
Oct 30 22:04:37.471004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:37.470969    5031 generic.go:356] "Generic (PLEG): container finished" podID="d4994728-c514-4fbc-a183-61eb70d1648c" containerID="932c63de613e1e15f1f730f51c0b305e279b5a95ef2d32e72ae7cc6e8a937b56" exitCode=0
Oct 30 22:04:37.471477 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:37.471134    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx" event={"ID":"d4994728-c514-4fbc-a183-61eb70d1648c","Type":"ContainerDied","Data":"932c63de613e1e15f1f730f51c0b305e279b5a95ef2d32e72ae7cc6e8a937b56"}
Oct 30 22:04:37.887920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:37.887863    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:38.654237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.643813    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:38.739667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.739570    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-xks-ca\") pod \"d4994728-c514-4fbc-a183-61eb70d1648c\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") "
Oct 30 22:04:38.739667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.739624    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4994728-c514-4fbc-a183-61eb70d1648c-kube-api-access-st7mg\" (UniqueName: \"kubernetes.io/projected/d4994728-c514-4fbc-a183-61eb70d1648c-kube-api-access-st7mg\") pod \"d4994728-c514-4fbc-a183-61eb70d1648c\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") "
Oct 30 22:04:38.739933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.739676    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-openshift-ca\") pod \"d4994728-c514-4fbc-a183-61eb70d1648c\" (UID: \"d4994728-c514-4fbc-a183-61eb70d1648c\") "
Oct 30 22:04:38.740364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.740335    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-openshift-ca" pod "d4994728-c514-4fbc-a183-61eb70d1648c" (UID: "d4994728-c514-4fbc-a183-61eb70d1648c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:38.743241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.742892    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-xks-ca" pod "d4994728-c514-4fbc-a183-61eb70d1648c" (UID: "d4994728-c514-4fbc-a183-61eb70d1648c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:38.743421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.743400    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4994728-c514-4fbc-a183-61eb70d1648c-kube-api-access-st7mg" pod "d4994728-c514-4fbc-a183-61eb70d1648c" (UID: "d4994728-c514-4fbc-a183-61eb70d1648c"). InnerVolumeSpecName "kube-api-access-st7mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:38.841823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.841787    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:38.842088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.842065    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7mg\" (UniqueName: \"kubernetes.io/projected/d4994728-c514-4fbc-a183-61eb70d1648c-kube-api-access-st7mg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:38.842277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:38.842261    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4994728-c514-4fbc-a183-61eb70d1648c-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:39.488542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:39.488512    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"
Oct 30 22:04:39.488542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:39.488524    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx" event={"ID":"d4994728-c514-4fbc-a183-61eb70d1648c","Type":"ContainerDied","Data":"7a9f39b246d84c7abd35d569ad649ec4f192f7257b4dc6fc63590afe02ea5ed7"}
Oct 30 22:04:39.488809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:39.488565    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9f39b246d84c7abd35d569ad649ec4f192f7257b4dc6fc63590afe02ea5ed7"
Oct 30 22:04:41.345880 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.345847    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"]
Oct 30 22:04:41.346486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.346470    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d4994728-c514-4fbc-a183-61eb70d1648c" containerName="revoke-keys"
Oct 30 22:04:41.346540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.346487    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4994728-c514-4fbc-a183-61eb70d1648c" containerName="revoke-keys"
Oct 30 22:04:41.346598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.346590    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4994728-c514-4fbc-a183-61eb70d1648c" containerName="revoke-keys"
Oct 30 22:04:41.350322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.350301    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.409904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.409871    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"]
Oct 30 22:04:41.455870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.455838    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:41.455870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.455850    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:41.455870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.455850    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:41.456111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.456003    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:04:41.476831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.476799    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhhw\" (UniqueName: \"kubernetes.io/projected/44a42928-be4d-41a9-9060-709d1fb76621-kube-api-access-pzhhw\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.476986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.476849    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.477033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.477019    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.578450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.578410    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.578669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.578511    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhhw\" (UniqueName: \"kubernetes.io/projected/44a42928-be4d-41a9-9060-709d1fb76621-kube-api-access-pzhhw\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.578669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.578533    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.579245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.579197    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.581029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.581009    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.643057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.642980    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhhw\" (UniqueName: \"kubernetes.io/projected/44a42928-be4d-41a9-9060-709d1fb76621-kube-api-access-pzhhw\") pod \"maas-api-revoke-sub-62f0739f-xxz7x\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:41.664877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:41.664845    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:42.048847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:42.048815    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"]
Oct 30 22:04:42.049923 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:04:42.049892    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a42928_be4d_41a9_9060_709d1fb76621.slice/crio-bc5751554620d645083a9f86b6480685ab15f5f169be4077fa9bb65453a70644 WatchSource:0}: Error finding container bc5751554620d645083a9f86b6480685ab15f5f169be4077fa9bb65453a70644: Status 404 returned error can't find the container with id bc5751554620d645083a9f86b6480685ab15f5f169be4077fa9bb65453a70644
Oct 30 22:04:42.512501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:42.512460    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x" event={"ID":"44a42928-be4d-41a9-9060-709d1fb76621","Type":"ContainerStarted","Data":"6723413e528802fa7e3bb2d89b86d300dfbb2ba191b08c52dc3979b90691e940"}
Oct 30 22:04:42.512974 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:42.512527    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x" event={"ID":"44a42928-be4d-41a9-9060-709d1fb76621","Type":"ContainerStarted","Data":"bc5751554620d645083a9f86b6480685ab15f5f169be4077fa9bb65453a70644"}
Oct 30 22:04:42.635129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:42.635066    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x" podStartSLOduration=1.635051849 podStartE2EDuration="1.635051849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.632349812 +0000 UTC m=+894.156492610" watchObservedRunningTime="2026-10-30 22:04:42.635051849 +0000 UTC m=+894.159194613"
Oct 30 22:04:42.716520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:42.716468    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:43.520712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:43.520675    5031 generic.go:356] "Generic (PLEG): container finished" podID="44a42928-be4d-41a9-9060-709d1fb76621" containerID="6723413e528802fa7e3bb2d89b86d300dfbb2ba191b08c52dc3979b90691e940" exitCode=0
Oct 30 22:04:43.521140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:43.520745    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x" event={"ID":"44a42928-be4d-41a9-9060-709d1fb76621","Type":"ContainerDied","Data":"6723413e528802fa7e3bb2d89b86d300dfbb2ba191b08c52dc3979b90691e940"}
Oct 30 22:04:43.711257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:43.711144    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:44.705638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.705605    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:44.815612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.815514    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a42928-be4d-41a9-9060-709d1fb76621-kube-api-access-pzhhw\" (UniqueName: \"kubernetes.io/projected/44a42928-be4d-41a9-9060-709d1fb76621-kube-api-access-pzhhw\") pod \"44a42928-be4d-41a9-9060-709d1fb76621\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") "
Oct 30 22:04:44.815612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.815569    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-xks-ca\") pod \"44a42928-be4d-41a9-9060-709d1fb76621\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") "
Oct 30 22:04:44.815887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.815759    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-openshift-ca\") pod \"44a42928-be4d-41a9-9060-709d1fb76621\" (UID: \"44a42928-be4d-41a9-9060-709d1fb76621\") "
Oct 30 22:04:44.816145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.816116    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-openshift-ca" pod "44a42928-be4d-41a9-9060-709d1fb76621" (UID: "44a42928-be4d-41a9-9060-709d1fb76621"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:44.816450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.816325    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:44.817848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.817819    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a42928-be4d-41a9-9060-709d1fb76621-kube-api-access-pzhhw" pod "44a42928-be4d-41a9-9060-709d1fb76621" (UID: "44a42928-be4d-41a9-9060-709d1fb76621"). InnerVolumeSpecName "kube-api-access-pzhhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:44.817954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.817868    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-xks-ca" pod "44a42928-be4d-41a9-9060-709d1fb76621" (UID: "44a42928-be4d-41a9-9060-709d1fb76621"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:44.917911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.917876    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhhw\" (UniqueName: \"kubernetes.io/projected/44a42928-be4d-41a9-9060-709d1fb76621-kube-api-access-pzhhw\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:44.917911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:44.917908    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a42928-be4d-41a9-9060-709d1fb76621-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:45.537785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:45.537755    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"
Oct 30 22:04:45.537994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:45.537796    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x" event={"ID":"44a42928-be4d-41a9-9060-709d1fb76621","Type":"ContainerDied","Data":"bc5751554620d645083a9f86b6480685ab15f5f169be4077fa9bb65453a70644"}
Oct 30 22:04:45.537994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:45.537843    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5751554620d645083a9f86b6480685ab15f5f169be4077fa9bb65453a70644"
Oct 30 22:04:46.133134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.133093    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"]
Oct 30 22:04:46.133916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.133894    5031 cpu_manager.go:414] "containerMap: removing container" podUID="44a42928-be4d-41a9-9060-709d1fb76621" containerName="revoke-keys"
Oct 30 22:04:46.133916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.133915    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a42928-be4d-41a9-9060-709d1fb76621" containerName="revoke-keys"
Oct 30 22:04:46.134122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.134109    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a42928-be4d-41a9-9060-709d1fb76621" containerName="revoke-keys"
Oct 30 22:04:46.138024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.137991    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.192759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.192728    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2182a5-openshift-default-dockercfg-ssq65\""
Oct 30 22:04:46.233747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.233713    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.233938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.233773    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.234005 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.233969    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.234076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.234010    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40284c30-15d8-4040-a338-c77efbdd8e81-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.234603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.234579    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40284c30-15d8-4040-a338-c77efbdd8e81-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.234680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.234634    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.234744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.234695    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkz9\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-kube-api-access-cpkz9\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.234744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.234735    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.234850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.234784    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.247015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.246988    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"]
Oct 30 22:04:46.336328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336298    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336343    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336407    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336434    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40284c30-15d8-4040-a338-c77efbdd8e81-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336456    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40284c30-15d8-4040-a338-c77efbdd8e81-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336480    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336528    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkz9\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-kube-api-access-cpkz9\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336565    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.336793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336608    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.337022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.336898    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.337329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.337272    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.337329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.337299    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40284c30-15d8-4040-a338-c77efbdd8e81-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.337481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.337380    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.337481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.337421    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.339178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.339155    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.339903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.339885    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40284c30-15d8-4040-a338-c77efbdd8e81-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.400658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.400591    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkz9\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-kube-api-access-cpkz9\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.401689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.401673    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.453820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.453795    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:46.915812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.915769    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"]
Oct 30 22:04:46.916385 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:04:46.916362    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40284c30_15d8_4040_a338_c77efbdd8e81.slice/crio-801e2a5ef32d68329c3be43a2d2e153e4339da06521c52819ccb3da7fb0c7088 WatchSource:0}: Error finding container 801e2a5ef32d68329c3be43a2d2e153e4339da06521c52819ccb3da7fb0c7088: Status 404 returned error can't find the container with id 801e2a5ef32d68329c3be43a2d2e153e4339da06521c52819ccb3da7fb0c7088
Oct 30 22:04:46.919560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.919528    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:04:46.919640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.919592    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:04:46.919640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:46.919621    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:04:47.557907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:47.557871    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" event={"ID":"40284c30-15d8-4040-a338-c77efbdd8e81","Type":"ContainerStarted","Data":"72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370"}
Oct 30 22:04:47.558384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:47.557954    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" event={"ID":"40284c30-15d8-4040-a338-c77efbdd8e81","Type":"ContainerStarted","Data":"801e2a5ef32d68329c3be43a2d2e153e4339da06521c52819ccb3da7fb0c7088"}
Oct 30 22:04:47.832288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:47.831120    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:48.466284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:48.459940    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:48.490234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:48.478327    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:48.490234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:48.478406    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:04:49.150236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:49.150049    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:04:49.163753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:49.163482    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:04:49.458661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:49.458428    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:49.458661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:49.458506    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:04:50.467285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:50.466621    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:50.467285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:50.466686    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:04:50.603747 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:04:50.603703    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:04:51.459540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:51.459504    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:51.459856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:51.459825    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:04:52.454612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.454567    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:52.455157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.454635    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:04:52.535235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.534739    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podStartSLOduration=6.534719825 podStartE2EDuration="6.534719825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:47.737377046 +0000 UTC m=+899.261519828" watchObservedRunningTime="2026-10-30 22:04:52.534719825 +0000 UTC m=+904.058862594"
Oct 30 22:04:52.539167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.539131    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"]
Oct 30 22:04:52.544030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.544013    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.599464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.599436    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"]
Oct 30 22:04:52.638254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.632555    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:52.638254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.632633    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:52.638254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.632673    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:52.644071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.644034    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fjb\" (UniqueName: \"kubernetes.io/projected/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-kube-api-access-76fjb\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.644238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.644180    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.644322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.644237    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.645776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.645753    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:04:52.745844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.745742    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.745844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.745788    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.746037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.745946    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fjb\" (UniqueName: \"kubernetes.io/projected/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-kube-api-access-76fjb\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.746622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.746591    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.749415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.749223    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.825240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.822061    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fjb\" (UniqueName: \"kubernetes.io/projected/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-kube-api-access-76fjb\") pod \"maas-api-revoke-sub-72f04383-592z9\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:52.859125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:52.859092    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:53.107142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.107102    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:53.281885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.281853    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"]
Oct 30 22:04:53.282399 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:04:53.282369    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7924d7a1_c1ba_41f2_aa1f_1b7adcef9cde.slice/crio-734e541708a75d887f00582cdb38b835626dc1d0a7d004a8107a1a6f90a3c576 WatchSource:0}: Error finding container 734e541708a75d887f00582cdb38b835626dc1d0a7d004a8107a1a6f90a3c576: Status 404 returned error can't find the container with id 734e541708a75d887f00582cdb38b835626dc1d0a7d004a8107a1a6f90a3c576
Oct 30 22:04:53.443040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.442962    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"]
Oct 30 22:04:53.448253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.448235    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.454494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.454461    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:04:53.454629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.454515    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:04:53.506511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.506478    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"]
Oct 30 22:04:53.556091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.556055    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwnb\" (UniqueName: \"kubernetes.io/projected/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-kube-api-access-2xwnb\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.556313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.556242    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.556384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.556341    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.657783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.657746    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.657997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.657846    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.657997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.657912    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwnb\" (UniqueName: \"kubernetes.io/projected/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-kube-api-access-2xwnb\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.658797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.658756    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.660517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.660486    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.714404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.714318    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9" event={"ID":"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde","Type":"ContainerStarted","Data":"df2c5a8a55551d03c64bae35c84cf751c6d92248f4b41360eec8340cc9690010"}
Oct 30 22:04:53.714404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.714385    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9" event={"ID":"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde","Type":"ContainerStarted","Data":"734e541708a75d887f00582cdb38b835626dc1d0a7d004a8107a1a6f90a3c576"}
Oct 30 22:04:53.722518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.722492    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwnb\" (UniqueName: \"kubernetes.io/projected/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-kube-api-access-2xwnb\") pod \"maas-api-revoke-sub-6488f356-87gpb\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.768050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.768021    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:53.847272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:53.845990    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9" podStartSLOduration=1.845969545 podStartE2EDuration="1.845969545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:53.840671732 +0000 UTC m=+905.364814498" watchObservedRunningTime="2026-10-30 22:04:53.845969545 +0000 UTC m=+905.370112311"
Oct 30 22:04:54.229250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:54.228367    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"]
Oct 30 22:04:54.460352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:54.460323    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:54.460715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:54.460687    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:54.461555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:54.461536    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:04:54.723196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:54.723165    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb" event={"ID":"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9","Type":"ContainerStarted","Data":"f46458ce12055bd61ca5401486e81b1a31edfc2c2d079be4bb0793f01f3a1a24"}
Oct 30 22:04:54.725282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:54.725256    5031 generic.go:356] "Generic (PLEG): container finished" podID="7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" containerID="df2c5a8a55551d03c64bae35c84cf751c6d92248f4b41360eec8340cc9690010" exitCode=0
Oct 30 22:04:54.725372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:54.725334    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9" event={"ID":"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde","Type":"ContainerDied","Data":"df2c5a8a55551d03c64bae35c84cf751c6d92248f4b41360eec8340cc9690010"}
Oct 30 22:04:55.734441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:55.734403    5031 generic.go:356] "Generic (PLEG): container finished" podID="70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" containerID="61735f08f852649ace0a5faa7ea7efed8033084f76ab45359caa612c2dc81daf" exitCode=0
Oct 30 22:04:55.734840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:55.734492    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb" event={"ID":"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9","Type":"ContainerDied","Data":"61735f08f852649ace0a5faa7ea7efed8033084f76ab45359caa612c2dc81daf"}
Oct 30 22:04:55.884410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:55.884382    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:56.014257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.014144    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-kube-api-access-76fjb\" (UniqueName: \"kubernetes.io/projected/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-kube-api-access-76fjb\") pod \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") "
Oct 30 22:04:56.014418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.014292    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-xks-ca\") pod \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") "
Oct 30 22:04:56.014418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.014409    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-openshift-ca\") pod \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\" (UID: \"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde\") "
Oct 30 22:04:56.014776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.014750    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-openshift-ca" pod "7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" (UID: "7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:56.014927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.014867    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:56.017171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.017146    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-xks-ca" pod "7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" (UID: "7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:56.017292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.017227    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-kube-api-access-76fjb" pod "7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" (UID: "7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde"). InnerVolumeSpecName "kube-api-access-76fjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:56.116313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.116279    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fjb\" (UniqueName: \"kubernetes.io/projected/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-kube-api-access-76fjb\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:56.116313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.116308    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:56.742371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.742341    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"
Oct 30 22:04:56.742815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.742379    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9" event={"ID":"7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde","Type":"ContainerDied","Data":"734e541708a75d887f00582cdb38b835626dc1d0a7d004a8107a1a6f90a3c576"}
Oct 30 22:04:56.742815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.742433    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734e541708a75d887f00582cdb38b835626dc1d0a7d004a8107a1a6f90a3c576"
Oct 30 22:04:56.902682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:56.902655    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:57.026809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.026730    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-xks-ca\") pod \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") "
Oct 30 22:04:57.026809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.026801    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-kube-api-access-2xwnb\" (UniqueName: \"kubernetes.io/projected/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-kube-api-access-2xwnb\") pod \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") "
Oct 30 22:04:57.026972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.026869    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-openshift-ca\") pod \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\" (UID: \"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9\") "
Oct 30 22:04:57.027694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.027653    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-openshift-ca" pod "70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" (UID: "70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:57.030112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.030084    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-xks-ca" pod "70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" (UID: "70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:57.030405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.030385    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-kube-api-access-2xwnb" pod "70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" (UID: "70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9"). InnerVolumeSpecName "kube-api-access-2xwnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:57.128782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.128752    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:57.128782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.128783    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwnb\" (UniqueName: \"kubernetes.io/projected/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-kube-api-access-2xwnb\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:57.128976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.128794    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:57.750740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.750707    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"
Oct 30 22:04:57.751159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.750732    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb" event={"ID":"70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9","Type":"ContainerDied","Data":"f46458ce12055bd61ca5401486e81b1a31edfc2c2d079be4bb0793f01f3a1a24"}
Oct 30 22:04:57.751159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:57.750790    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46458ce12055bd61ca5401486e81b1a31edfc2c2d079be4bb0793f01f3a1a24"
Oct 30 22:04:58.446141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:04:58.446070    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:00.374666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.374629    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"]
Oct 30 22:05:00.375339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.375319    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" containerName="revoke-keys"
Oct 30 22:05:00.375339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.375336    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" containerName="revoke-keys"
Oct 30 22:05:00.375516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.375358    5031 cpu_manager.go:414] "containerMap: removing container" podUID="70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" containerName="revoke-keys"
Oct 30 22:05:00.375516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.375363    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" containerName="revoke-keys"
Oct 30 22:05:00.375516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.375465    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" containerName="revoke-keys"
Oct 30 22:05:00.375516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.375480    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" containerName="revoke-keys"
Oct 30 22:05:00.379104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.379088    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.441021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.440983    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"]
Oct 30 22:05:00.467262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.467191    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.467414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.467320    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tzp\" (UniqueName: \"kubernetes.io/projected/d39ad791-00d3-4538-8b2a-420e184c5fdd-kube-api-access-v6tzp\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.467414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.467355    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.485391 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.485367    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:00.485537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.485405    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:00.485537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.485435    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:00.485537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.485509    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:00.568221 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.568181    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tzp\" (UniqueName: \"kubernetes.io/projected/d39ad791-00d3-4538-8b2a-420e184c5fdd-kube-api-access-v6tzp\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.568410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.568247    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.568410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.568398    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.569041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.569022    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.570969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.570945    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.631742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.631679    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tzp\" (UniqueName: \"kubernetes.io/projected/d39ad791-00d3-4538-8b2a-420e184c5fdd-kube-api-access-v6tzp\") pod \"maas-api-revoke-sub-805ef94f-cbxq7\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:00.693764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:00.693735    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:01.090886 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:01.090855    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39ad791_00d3_4538_8b2a_420e184c5fdd.slice/crio-0a27e4ded189d9841a4931921065bdbd9788ffc72219457b99eb19c7786abbce WatchSource:0}: Error finding container 0a27e4ded189d9841a4931921065bdbd9788ffc72219457b99eb19c7786abbce: Status 404 returned error can't find the container with id 0a27e4ded189d9841a4931921065bdbd9788ffc72219457b99eb19c7786abbce
Oct 30 22:05:01.091960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:01.091935    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"]
Oct 30 22:05:01.787580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:01.787467    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7" event={"ID":"d39ad791-00d3-4538-8b2a-420e184c5fdd","Type":"ContainerStarted","Data":"5e9b02ab3b96e4ae78800453160dad9fbd27d0a1575b00789af8348a6a6c70b9"}
Oct 30 22:05:01.787580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:01.787568    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7" event={"ID":"d39ad791-00d3-4538-8b2a-420e184c5fdd","Type":"ContainerStarted","Data":"0a27e4ded189d9841a4931921065bdbd9788ffc72219457b99eb19c7786abbce"}
Oct 30 22:05:01.912586 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:01.912532    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7" podStartSLOduration=1.912516267 podStartE2EDuration="1.912516267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:01.908361465 +0000 UTC m=+913.432504241" watchObservedRunningTime="2026-10-30 22:05:01.912516267 +0000 UTC m=+913.436659031"
Oct 30 22:05:02.796390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:02.796346    5031 generic.go:356] "Generic (PLEG): container finished" podID="d39ad791-00d3-4538-8b2a-420e184c5fdd" containerID="5e9b02ab3b96e4ae78800453160dad9fbd27d0a1575b00789af8348a6a6c70b9" exitCode=0
Oct 30 22:05:02.796880 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:02.796471    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7" event={"ID":"d39ad791-00d3-4538-8b2a-420e184c5fdd","Type":"ContainerDied","Data":"5e9b02ab3b96e4ae78800453160dad9fbd27d0a1575b00789af8348a6a6c70b9"}
Oct 30 22:05:03.516282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:03.516244    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:03.859028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:03.858988    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:05:03.992196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:03.992170    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:04.117064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.117033    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39ad791-00d3-4538-8b2a-420e184c5fdd-kube-api-access-v6tzp\" (UniqueName: \"kubernetes.io/projected/d39ad791-00d3-4538-8b2a-420e184c5fdd-kube-api-access-v6tzp\") pod \"d39ad791-00d3-4538-8b2a-420e184c5fdd\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") "
Oct 30 22:05:04.117274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.117153    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-openshift-ca\") pod \"d39ad791-00d3-4538-8b2a-420e184c5fdd\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") "
Oct 30 22:05:04.117274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.117257    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-xks-ca\") pod \"d39ad791-00d3-4538-8b2a-420e184c5fdd\" (UID: \"d39ad791-00d3-4538-8b2a-420e184c5fdd\") "
Oct 30 22:05:04.117536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.117512    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-openshift-ca" pod "d39ad791-00d3-4538-8b2a-420e184c5fdd" (UID: "d39ad791-00d3-4538-8b2a-420e184c5fdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:04.117706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.117692    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:04.119434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.119412    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-xks-ca" pod "d39ad791-00d3-4538-8b2a-420e184c5fdd" (UID: "d39ad791-00d3-4538-8b2a-420e184c5fdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:04.119737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.119722    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39ad791-00d3-4538-8b2a-420e184c5fdd-kube-api-access-v6tzp" pod "d39ad791-00d3-4538-8b2a-420e184c5fdd" (UID: "d39ad791-00d3-4538-8b2a-420e184c5fdd"). InnerVolumeSpecName "kube-api-access-v6tzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:04.225091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.225053    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39ad791-00d3-4538-8b2a-420e184c5fdd-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:04.225091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.225087    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tzp\" (UniqueName: \"kubernetes.io/projected/d39ad791-00d3-4538-8b2a-420e184c5fdd-kube-api-access-v6tzp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:04.816909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.816855    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7" event={"ID":"d39ad791-00d3-4538-8b2a-420e184c5fdd","Type":"ContainerDied","Data":"0a27e4ded189d9841a4931921065bdbd9788ffc72219457b99eb19c7786abbce"}
Oct 30 22:05:04.817452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.817429    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a27e4ded189d9841a4931921065bdbd9788ffc72219457b99eb19c7786abbce"
Oct 30 22:05:04.817643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:04.817242    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"
Oct 30 22:05:05.853843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.853808    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"]
Oct 30 22:05:05.854485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.854469    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d39ad791-00d3-4538-8b2a-420e184c5fdd" containerName="revoke-keys"
Oct 30 22:05:05.854540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.854486    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39ad791-00d3-4538-8b2a-420e184c5fdd" containerName="revoke-keys"
Oct 30 22:05:05.854605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.854595    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39ad791-00d3-4538-8b2a-420e184c5fdd" containerName="revoke-keys"
Oct 30 22:05:05.858621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.858605    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.947386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.947353    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.947567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.947396    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.947567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.947545    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c671a13e-3166-4293-a989-956f2f6f2873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.947696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.947585    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.947814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.947757    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c671a13e-3166-4293-a989-956f2f6f2873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.948077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.948048    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.948191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.948121    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.948191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.948154    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ll9g\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-kube-api-access-4ll9g\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.948313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.948199    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:05.987042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:05.987006    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"]
Oct 30 22:05:06.049917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.049881    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.049917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.049918    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.049967    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c671a13e-3166-4293-a989-956f2f6f2873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050001    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050062    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c671a13e-3166-4293-a989-956f2f6f2873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050102    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050151    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050182    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ll9g\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-kube-api-access-4ll9g\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050267    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050333    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050431    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050735    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.050952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.050928    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c671a13e-3166-4293-a989-956f2f6f2873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.051047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.051027    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.052666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.052644    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.052989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.052969    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c671a13e-3166-4293-a989-956f2f6f2873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.118833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.118762    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.118957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.118831    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ll9g\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-kube-api-access-4ll9g\") pod \"maas-default-gateway-openshift-default-67bf986f99-7k7bp\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.173827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.173795    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:06.620622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.620587    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"]
Oct 30 22:05:06.621172 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:06.621140    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc671a13e_3166_4293_a989_956f2f6f2873.slice/crio-afa799073b95e80ea92956d7ca2290b6a42b844ba377c05228eea5333353f4dd WatchSource:0}: Error finding container afa799073b95e80ea92956d7ca2290b6a42b844ba377c05228eea5333353f4dd: Status 404 returned error can't find the container with id afa799073b95e80ea92956d7ca2290b6a42b844ba377c05228eea5333353f4dd
Oct 30 22:05:06.623992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.623955    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.624090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.624028    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.624090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.624059    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:06.835401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.835367    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" event={"ID":"c671a13e-3166-4293-a989-956f2f6f2873","Type":"ContainerStarted","Data":"6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d"}
Oct 30 22:05:06.835583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.835426    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" event={"ID":"c671a13e-3166-4293-a989-956f2f6f2873","Type":"ContainerStarted","Data":"afa799073b95e80ea92956d7ca2290b6a42b844ba377c05228eea5333353f4dd"}
Oct 30 22:05:06.964468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:06.964330    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podStartSLOduration=1.964311074 podStartE2EDuration="1.964311074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:06.961769005 +0000 UTC m=+918.485911788" watchObservedRunningTime="2026-10-30 22:05:06.964311074 +0000 UTC m=+918.488453840"
Oct 30 22:05:07.174948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:07.174904    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:07.176595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:07.176566    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:07.176823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:07.176793    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:08.174535 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:08.174501    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:08.175009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:08.174560    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:08.986703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:08.986664    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:09.174488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:09.174446    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:09.174685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:09.174508    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:09.939860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:09.939817    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"]
Oct 30 22:05:09.945708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:09.945679    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:09.995633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:09.995604    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"]
Oct 30 22:05:09.997912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:09.997886    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/a72911b4-60ba-41ea-a998-be27d168c586-kube-api-access-gjrb7\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.001472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.001442    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.001665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.001646    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.040924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.040891    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:10.041473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.041316    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:10.041685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.041403    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:10.054280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.052479    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:10.103193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.103161    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.103401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.103284    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.103401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.103357    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/a72911b4-60ba-41ea-a998-be27d168c586-kube-api-access-gjrb7\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.103914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.103886    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.105798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.105781    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.167141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.167105    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/a72911b4-60ba-41ea-a998-be27d168c586-kube-api-access-gjrb7\") pod \"maas-api-revoke-sub-e434794e-sjltg\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.174624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.174594    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:10.174758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.174680    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:10.244698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.244603    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"]
Oct 30 22:05:10.250191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.250168    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.263374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.263352    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:10.329464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.329416    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"]
Oct 30 22:05:10.339254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.334641    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.339254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.334710    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzlx\" (UniqueName: \"kubernetes.io/projected/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-kube-api-access-ltzlx\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.339254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.334919    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.445487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.444786    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.445487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.444916    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.445487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.444965    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzlx\" (UniqueName: \"kubernetes.io/projected/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-kube-api-access-ltzlx\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.446114 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.446080    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.465632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.463510    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"]
Oct 30 22:05:10.465632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.465194    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.470445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.470422    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.491651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.491622    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"]
Oct 30 22:05:10.547196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.547087    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzlx\" (UniqueName: \"kubernetes.io/projected/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-kube-api-access-ltzlx\") pod \"maas-api-revoke-sub-2b15fd75-fnlqv\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.547945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.547914    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.548067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.547980    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.548067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.548061    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdg8r\" (UniqueName: \"kubernetes.io/projected/47fd3610-6642-485b-9c49-4eb2b2912eee-kube-api-access-xdg8r\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.565596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.565540    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:10.665848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.665806    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdg8r\" (UniqueName: \"kubernetes.io/projected/47fd3610-6642-485b-9c49-4eb2b2912eee-kube-api-access-xdg8r\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.666055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.666035    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.666139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.666093    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.667291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.667266    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.669758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.669731    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.705815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.705782    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"]
Oct 30 22:05:10.727234 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:10.715542    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72911b4_60ba_41ea_a998_be27d168c586.slice/crio-aa55ac053eb91ae05dacec36c92d857a82ebb170f03689aeb379f6b61b377c47 WatchSource:0}: Error finding container aa55ac053eb91ae05dacec36c92d857a82ebb170f03689aeb379f6b61b377c47: Status 404 returned error can't find the container with id aa55ac053eb91ae05dacec36c92d857a82ebb170f03689aeb379f6b61b377c47
Oct 30 22:05:10.745643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.745512    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdg8r\" (UniqueName: \"kubernetes.io/projected/47fd3610-6642-485b-9c49-4eb2b2912eee-kube-api-access-xdg8r\") pod \"maas-api-revoke-sub-9984e673-zz2xw\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.795177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.795153    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:10.877705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:10.877670    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg" event={"ID":"a72911b4-60ba-41ea-a998-be27d168c586","Type":"ContainerStarted","Data":"aa55ac053eb91ae05dacec36c92d857a82ebb170f03689aeb379f6b61b377c47"}
Oct 30 22:05:11.000828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.000801    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"]
Oct 30 22:05:11.174761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.174725    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:11.175150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.174772    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:11.340403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.340365    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"]
Oct 30 22:05:11.346935 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:11.346906    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fd3610_6642_485b_9c49_4eb2b2912eee.slice/crio-078775fcf881324919ac97f54d70bc92376fad981841536a7a5c158505faec37 WatchSource:0}: Error finding container 078775fcf881324919ac97f54d70bc92376fad981841536a7a5c158505faec37: Status 404 returned error can't find the container with id 078775fcf881324919ac97f54d70bc92376fad981841536a7a5c158505faec37
Oct 30 22:05:11.895291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.895257    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw" event={"ID":"47fd3610-6642-485b-9c49-4eb2b2912eee","Type":"ContainerStarted","Data":"00fe5c4cb811463da1c1fceb5760a433a6a410b4a180cbda50233e8f885b60dd"}
Oct 30 22:05:11.895555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.895536    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw" event={"ID":"47fd3610-6642-485b-9c49-4eb2b2912eee","Type":"ContainerStarted","Data":"078775fcf881324919ac97f54d70bc92376fad981841536a7a5c158505faec37"}
Oct 30 22:05:11.897399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.897374    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv" event={"ID":"f65d9e8d-8440-4322-ab04-cfe33c48f1f5","Type":"ContainerStarted","Data":"e96899cd0669367ee17e5cfa356068d4c3fdf9dd6dcd46b6aa1f9f9d72220231"}
Oct 30 22:05:11.897528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.897414    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv" event={"ID":"f65d9e8d-8440-4322-ab04-cfe33c48f1f5","Type":"ContainerStarted","Data":"31886642abb1d399902335ac1f5a54fe6c5ed644c9af6208e62441970cd023fc"}
Oct 30 22:05:11.899034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:11.899012    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg" event={"ID":"a72911b4-60ba-41ea-a998-be27d168c586","Type":"ContainerStarted","Data":"bb936b5790b2c2c492bccb573d3c43d15e0d796fd1e1ad3a40ae4856f5585773"}
Oct 30 22:05:11.942603 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:05:11.942570    5031 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72911b4_60ba_41ea_a998_be27d168c586.slice/crio-conmon-bb936b5790b2c2c492bccb573d3c43d15e0d796fd1e1ad3a40ae4856f5585773.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:05:12.033176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.032839    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw" podStartSLOduration=2.032817932 podStartE2EDuration="2.032817932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:12.023645795 +0000 UTC m=+923.547788561" watchObservedRunningTime="2026-10-30 22:05:12.032817932 +0000 UTC m=+923.556960698"
Oct 30 22:05:12.144982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.144927    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg" podStartSLOduration=3.14491257 podStartE2EDuration="3.14491257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:12.144089958 +0000 UTC m=+923.668232747" watchObservedRunningTime="2026-10-30 22:05:12.14491257 +0000 UTC m=+923.669055334"
Oct 30 22:05:12.174805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.174763    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:12.175309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.174824    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:12.266416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.266356    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv" podStartSLOduration=2.266338419 podStartE2EDuration="2.266338419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:12.265414554 +0000 UTC m=+923.789557318" watchObservedRunningTime="2026-10-30 22:05:12.266338419 +0000 UTC m=+923.790481184"
Oct 30 22:05:12.909103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.909067    5031 generic.go:356] "Generic (PLEG): container finished" podID="47fd3610-6642-485b-9c49-4eb2b2912eee" containerID="00fe5c4cb811463da1c1fceb5760a433a6a410b4a180cbda50233e8f885b60dd" exitCode=0
Oct 30 22:05:12.909310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.909145    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw" event={"ID":"47fd3610-6642-485b-9c49-4eb2b2912eee","Type":"ContainerDied","Data":"00fe5c4cb811463da1c1fceb5760a433a6a410b4a180cbda50233e8f885b60dd"}
Oct 30 22:05:12.911123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.911097    5031 generic.go:356] "Generic (PLEG): container finished" podID="f65d9e8d-8440-4322-ab04-cfe33c48f1f5" containerID="e96899cd0669367ee17e5cfa356068d4c3fdf9dd6dcd46b6aa1f9f9d72220231" exitCode=0
Oct 30 22:05:12.911284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.911175    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv" event={"ID":"f65d9e8d-8440-4322-ab04-cfe33c48f1f5","Type":"ContainerDied","Data":"e96899cd0669367ee17e5cfa356068d4c3fdf9dd6dcd46b6aa1f9f9d72220231"}
Oct 30 22:05:12.913055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.913034    5031 generic.go:356] "Generic (PLEG): container finished" podID="a72911b4-60ba-41ea-a998-be27d168c586" containerID="bb936b5790b2c2c492bccb573d3c43d15e0d796fd1e1ad3a40ae4856f5585773" exitCode=0
Oct 30 22:05:12.913158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:12.913091    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg" event={"ID":"a72911b4-60ba-41ea-a998-be27d168c586","Type":"ContainerDied","Data":"bb936b5790b2c2c492bccb573d3c43d15e0d796fd1e1ad3a40ae4856f5585773"}
Oct 30 22:05:13.174564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:13.174483    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:13.174564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:13.174535    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:14.118222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.118174    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:14.174570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.174238    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:05:14.174570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.174297    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:05:14.206518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.206490    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-openshift-ca\") pod \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") "
Oct 30 22:05:14.206688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.206569    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-kube-api-access-ltzlx\" (UniqueName: \"kubernetes.io/projected/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-kube-api-access-ltzlx\") pod \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") "
Oct 30 22:05:14.206688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.206670    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-xks-ca\") pod \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\" (UID: \"f65d9e8d-8440-4322-ab04-cfe33c48f1f5\") "
Oct 30 22:05:14.208616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.208584    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-openshift-ca" pod "f65d9e8d-8440-4322-ab04-cfe33c48f1f5" (UID: "f65d9e8d-8440-4322-ab04-cfe33c48f1f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:14.209338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.209310    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-xks-ca" pod "f65d9e8d-8440-4322-ab04-cfe33c48f1f5" (UID: "f65d9e8d-8440-4322-ab04-cfe33c48f1f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:14.210824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.210801    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:14.211011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.210988    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-kube-api-access-ltzlx" pod "f65d9e8d-8440-4322-ab04-cfe33c48f1f5" (UID: "f65d9e8d-8440-4322-ab04-cfe33c48f1f5"). InnerVolumeSpecName "kube-api-access-ltzlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:14.226276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.226246    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:14.307423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.307387    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72911b4-60ba-41ea-a998-be27d168c586-kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/a72911b4-60ba-41ea-a998-be27d168c586-kube-api-access-gjrb7\") pod \"a72911b4-60ba-41ea-a998-be27d168c586\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") "
Oct 30 22:05:14.307632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.307605    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fd3610-6642-485b-9c49-4eb2b2912eee-kube-api-access-xdg8r\" (UniqueName: \"kubernetes.io/projected/47fd3610-6642-485b-9c49-4eb2b2912eee-kube-api-access-xdg8r\") pod \"47fd3610-6642-485b-9c49-4eb2b2912eee\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") "
Oct 30 22:05:14.307699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.307641    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-openshift-ca\") pod \"a72911b4-60ba-41ea-a998-be27d168c586\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") "
Oct 30 22:05:14.307761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.307701    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-xks-ca\") pod \"47fd3610-6642-485b-9c49-4eb2b2912eee\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") "
Oct 30 22:05:14.307825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.307757    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-xks-ca\") pod \"a72911b4-60ba-41ea-a998-be27d168c586\" (UID: \"a72911b4-60ba-41ea-a998-be27d168c586\") "
Oct 30 22:05:14.308046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.308015    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-openshift-ca" pod "a72911b4-60ba-41ea-a998-be27d168c586" (UID: "a72911b4-60ba-41ea-a998-be27d168c586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:14.308229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.308191    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-openshift-ca\") pod \"47fd3610-6642-485b-9c49-4eb2b2912eee\" (UID: \"47fd3610-6642-485b-9c49-4eb2b2912eee\") "
Oct 30 22:05:14.309302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.309253    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-openshift-ca" pod "47fd3610-6642-485b-9c49-4eb2b2912eee" (UID: "47fd3610-6642-485b-9c49-4eb2b2912eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:14.309420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.309360    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.309420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.309379    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.309420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.309393    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.309420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.309408    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzlx\" (UniqueName: \"kubernetes.io/projected/f65d9e8d-8440-4322-ab04-cfe33c48f1f5-kube-api-access-ltzlx\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.310859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.310822    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-xks-ca" pod "a72911b4-60ba-41ea-a998-be27d168c586" (UID: "a72911b4-60ba-41ea-a998-be27d168c586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:14.310969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.310876    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-xks-ca" pod "47fd3610-6642-485b-9c49-4eb2b2912eee" (UID: "47fd3610-6642-485b-9c49-4eb2b2912eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:14.311203 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.311176    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72911b4-60ba-41ea-a998-be27d168c586-kube-api-access-gjrb7" pod "a72911b4-60ba-41ea-a998-be27d168c586" (UID: "a72911b4-60ba-41ea-a998-be27d168c586"). InnerVolumeSpecName "kube-api-access-gjrb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:14.311420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.311403    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fd3610-6642-485b-9c49-4eb2b2912eee-kube-api-access-xdg8r" pod "47fd3610-6642-485b-9c49-4eb2b2912eee" (UID: "47fd3610-6642-485b-9c49-4eb2b2912eee"). InnerVolumeSpecName "kube-api-access-xdg8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:14.411229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.411169    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdg8r\" (UniqueName: \"kubernetes.io/projected/47fd3610-6642-485b-9c49-4eb2b2912eee-kube-api-access-xdg8r\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.411432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.411262    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.411432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.411281    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72911b4-60ba-41ea-a998-be27d168c586-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.411432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.411295    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47fd3610-6642-485b-9c49-4eb2b2912eee-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.411432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.411328    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrb7\" (UniqueName: \"kubernetes.io/projected/a72911b4-60ba-41ea-a998-be27d168c586-kube-api-access-gjrb7\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:14.591644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.591549    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:14.931411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.931374    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"
Oct 30 22:05:14.931411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.931394    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg" event={"ID":"a72911b4-60ba-41ea-a998-be27d168c586","Type":"ContainerDied","Data":"aa55ac053eb91ae05dacec36c92d857a82ebb170f03689aeb379f6b61b377c47"}
Oct 30 22:05:14.931706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.931472    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa55ac053eb91ae05dacec36c92d857a82ebb170f03689aeb379f6b61b377c47"
Oct 30 22:05:14.933173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.933147    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"
Oct 30 22:05:14.933173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.933156    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw" event={"ID":"47fd3610-6642-485b-9c49-4eb2b2912eee","Type":"ContainerDied","Data":"078775fcf881324919ac97f54d70bc92376fad981841536a7a5c158505faec37"}
Oct 30 22:05:14.933414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.933193    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="078775fcf881324919ac97f54d70bc92376fad981841536a7a5c158505faec37"
Oct 30 22:05:14.935067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.935043    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv" event={"ID":"f65d9e8d-8440-4322-ab04-cfe33c48f1f5","Type":"ContainerDied","Data":"31886642abb1d399902335ac1f5a54fe6c5ed644c9af6208e62441970cd023fc"}
Oct 30 22:05:14.935187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.935079    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31886642abb1d399902335ac1f5a54fe6c5ed644c9af6208e62441970cd023fc"
Oct 30 22:05:14.935187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:14.935085    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"
Oct 30 22:05:15.178787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:15.178755    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:15.179249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:15.179052    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:15.179914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:15.179894    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:05:16.752222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.752180    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"]
Oct 30 22:05:16.752843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.752827    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a72911b4-60ba-41ea-a998-be27d168c586" containerName="revoke-keys"
Oct 30 22:05:16.752909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.752845    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72911b4-60ba-41ea-a998-be27d168c586" containerName="revoke-keys"
Oct 30 22:05:16.752909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.752886    5031 cpu_manager.go:414] "containerMap: removing container" podUID="47fd3610-6642-485b-9c49-4eb2b2912eee" containerName="revoke-keys"
Oct 30 22:05:16.752909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.752893    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fd3610-6642-485b-9c49-4eb2b2912eee" containerName="revoke-keys"
Oct 30 22:05:16.752909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.752903    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f65d9e8d-8440-4322-ab04-cfe33c48f1f5" containerName="revoke-keys"
Oct 30 22:05:16.753061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.752910    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65d9e8d-8440-4322-ab04-cfe33c48f1f5" containerName="revoke-keys"
Oct 30 22:05:16.753061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.753014    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fd3610-6642-485b-9c49-4eb2b2912eee" containerName="revoke-keys"
Oct 30 22:05:16.753061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.753025    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65d9e8d-8440-4322-ab04-cfe33c48f1f5" containerName="revoke-keys"
Oct 30 22:05:16.753061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.753033    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72911b4-60ba-41ea-a998-be27d168c586" containerName="revoke-keys"
Oct 30 22:05:16.757795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.757774    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.819258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.819229    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"]
Oct 30 22:05:16.841850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.841821    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhghm\" (UniqueName: \"kubernetes.io/projected/8b69f820-692e-40ed-b295-3b7655bad054-kube-api-access-dhghm\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.842009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.841981    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.842099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.842086    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.864955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.864922    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:16.865194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.865174    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:16.865265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.865249    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:16.886702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.886676    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:16.943801 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.943768    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhghm\" (UniqueName: \"kubernetes.io/projected/8b69f820-692e-40ed-b295-3b7655bad054-kube-api-access-dhghm\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.943968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.943849    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.943968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.943916    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.944754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.944729    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:16.946607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:16.946585    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:17.008076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:17.008008    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhghm\" (UniqueName: \"kubernetes.io/projected/8b69f820-692e-40ed-b295-3b7655bad054-kube-api-access-dhghm\") pod \"maas-api-revoke-sub-8788fd31-qhccs\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:17.073100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:17.073066    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:17.461462 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:17.461431    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b69f820_692e_40ed_b295_3b7655bad054.slice/crio-efdaf9851d42a47139edfd40ddf20c9c798b740a78b7ae2904c0c5765331c2cd WatchSource:0}: Error finding container efdaf9851d42a47139edfd40ddf20c9c798b740a78b7ae2904c0c5765331c2cd: Status 404 returned error can't find the container with id efdaf9851d42a47139edfd40ddf20c9c798b740a78b7ae2904c0c5765331c2cd
Oct 30 22:05:17.462147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:17.462120    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"]
Oct 30 22:05:17.964804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:17.964765    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs" event={"ID":"8b69f820-692e-40ed-b295-3b7655bad054","Type":"ContainerStarted","Data":"74f44971c5ad96c6522295c1389fe25e1ed6f4144dffb0cfb02ae31a213a5914"}
Oct 30 22:05:17.965193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:17.964829    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs" event={"ID":"8b69f820-692e-40ed-b295-3b7655bad054","Type":"ContainerStarted","Data":"efdaf9851d42a47139edfd40ddf20c9c798b740a78b7ae2904c0c5765331c2cd"}
Oct 30 22:05:18.088282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:18.088223    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs" podStartSLOduration=2.088191495 podStartE2EDuration="2.088191495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:18.087181725 +0000 UTC m=+929.611324508" watchObservedRunningTime="2026-10-30 22:05:18.088191495 +0000 UTC m=+929.612334259"
Oct 30 22:05:18.974521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:18.974480    5031 generic.go:356] "Generic (PLEG): container finished" podID="8b69f820-692e-40ed-b295-3b7655bad054" containerID="74f44971c5ad96c6522295c1389fe25e1ed6f4144dffb0cfb02ae31a213a5914" exitCode=0
Oct 30 22:05:18.974985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:18.974571    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs" event={"ID":"8b69f820-692e-40ed-b295-3b7655bad054","Type":"ContainerDied","Data":"74f44971c5ad96c6522295c1389fe25e1ed6f4144dffb0cfb02ae31a213a5914"}
Oct 30 22:05:19.614367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:19.614328    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:20.187270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.187230    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:20.285661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.285571    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-xks-ca\") pod \"8b69f820-692e-40ed-b295-3b7655bad054\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") "
Oct 30 22:05:20.285823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.285691    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b69f820-692e-40ed-b295-3b7655bad054-kube-api-access-dhghm\" (UniqueName: \"kubernetes.io/projected/8b69f820-692e-40ed-b295-3b7655bad054-kube-api-access-dhghm\") pod \"8b69f820-692e-40ed-b295-3b7655bad054\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") "
Oct 30 22:05:20.285823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.285722    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-openshift-ca\") pod \"8b69f820-692e-40ed-b295-3b7655bad054\" (UID: \"8b69f820-692e-40ed-b295-3b7655bad054\") "
Oct 30 22:05:20.286113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.286088    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-openshift-ca" pod "8b69f820-692e-40ed-b295-3b7655bad054" (UID: "8b69f820-692e-40ed-b295-3b7655bad054"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:20.287913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.287889    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-xks-ca" pod "8b69f820-692e-40ed-b295-3b7655bad054" (UID: "8b69f820-692e-40ed-b295-3b7655bad054"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:20.288015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.287941    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b69f820-692e-40ed-b295-3b7655bad054-kube-api-access-dhghm" pod "8b69f820-692e-40ed-b295-3b7655bad054" (UID: "8b69f820-692e-40ed-b295-3b7655bad054"). InnerVolumeSpecName "kube-api-access-dhghm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:20.387048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.387017    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhghm\" (UniqueName: \"kubernetes.io/projected/8b69f820-692e-40ed-b295-3b7655bad054-kube-api-access-dhghm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:20.387048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.387046    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:20.387048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.387056    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b69f820-692e-40ed-b295-3b7655bad054-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:20.542337 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:05:20.542243    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:05:20.968662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.968628    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"]
Oct 30 22:05:20.969368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.969336    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8b69f820-692e-40ed-b295-3b7655bad054" containerName="revoke-keys"
Oct 30 22:05:20.969368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.969369    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69f820-692e-40ed-b295-3b7655bad054" containerName="revoke-keys"
Oct 30 22:05:20.969532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.969501    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69f820-692e-40ed-b295-3b7655bad054" containerName="revoke-keys"
Oct 30 22:05:20.973171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.973151    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:20.990635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.990605    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs" event={"ID":"8b69f820-692e-40ed-b295-3b7655bad054","Type":"ContainerDied","Data":"efdaf9851d42a47139edfd40ddf20c9c798b740a78b7ae2904c0c5765331c2cd"}
Oct 30 22:05:20.990635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.990626    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"
Oct 30 22:05:20.990775 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:20.990654    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efdaf9851d42a47139edfd40ddf20c9c798b740a78b7ae2904c0c5765331c2cd"
Oct 30 22:05:21.064192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.064137    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"]
Oct 30 22:05:21.070404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.070371    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.082917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.082885    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"]
Oct 30 22:05:21.094119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094091    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094133    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094171    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094228    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094324    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sl7k\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-kube-api-access-5sl7k\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094358    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094480    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094561 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.094595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.094580    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.128249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.128200    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"]
Oct 30 22:05:21.131000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.130973    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:21.131167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.131033    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:21.131341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.131189    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:21.131440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.131335    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:21.195669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195634    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195678    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195738    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4n8\" (UniqueName: \"kubernetes.io/projected/7b048e45-b26f-4a64-a26c-4d6eea108f2f-kube-api-access-nx4n8\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195762    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195809    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195852    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195887    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195903    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195932    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sl7k\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-kube-api-access-5sl7k\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.195980    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.196085    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.196123    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.196365    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.196392    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.196658    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.196859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.196838    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.197019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.196998    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.198197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.198176    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.198512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.198496    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.260440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.260361    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.260584 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.260555    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sl7k\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-kube-api-access-5sl7k\") pod \"maas-default-gateway-openshift-default-67bf986f99-rknl4\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.287996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.287965    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:21.297059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.297027    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.297170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.297113    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4n8\" (UniqueName: \"kubernetes.io/projected/7b048e45-b26f-4a64-a26c-4d6eea108f2f-kube-api-access-nx4n8\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.297170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.297146    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.297865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.297809    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.299701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.299678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.364671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.364634    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4n8\" (UniqueName: \"kubernetes.io/projected/7b048e45-b26f-4a64-a26c-4d6eea108f2f-kube-api-access-nx4n8\") pod \"maas-api-revoke-sub-03613f5b-4stxl\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.391288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.391257    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:21.733092 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:21.732913    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a7728b_68ff_4812_b8ea_e7ea9385e08c.slice/crio-d04ad3a2d8bab5a222015503deeba340e02451a46cf2a7f0c50b3e52a305c76f WatchSource:0}: Error finding container d04ad3a2d8bab5a222015503deeba340e02451a46cf2a7f0c50b3e52a305c76f: Status 404 returned error can't find the container with id d04ad3a2d8bab5a222015503deeba340e02451a46cf2a7f0c50b3e52a305c76f
Oct 30 22:05:21.733534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.733505    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"]
Oct 30 22:05:21.735841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.735798    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:21.735935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.735864    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:21.735935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.735913    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:21.784499 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:21.784445    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b048e45_b26f_4a64_a26c_4d6eea108f2f.slice/crio-638cb15c3e0ae930f79203489dd21d4c3d8f544c83cfa773e6579ec0d6dd6e14 WatchSource:0}: Error finding container 638cb15c3e0ae930f79203489dd21d4c3d8f544c83cfa773e6579ec0d6dd6e14: Status 404 returned error can't find the container with id 638cb15c3e0ae930f79203489dd21d4c3d8f544c83cfa773e6579ec0d6dd6e14
Oct 30 22:05:21.786542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:21.786511    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"]
Oct 30 22:05:22.001101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:22.000988    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl" event={"ID":"7b048e45-b26f-4a64-a26c-4d6eea108f2f","Type":"ContainerStarted","Data":"638cb15c3e0ae930f79203489dd21d4c3d8f544c83cfa773e6579ec0d6dd6e14"}
Oct 30 22:05:22.003080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:22.003047    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" event={"ID":"e5a7728b-68ff-4812-b8ea-e7ea9385e08c","Type":"ContainerStarted","Data":"bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0"}
Oct 30 22:05:22.003259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:22.003107    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" event={"ID":"e5a7728b-68ff-4812-b8ea-e7ea9385e08c","Type":"ContainerStarted","Data":"d04ad3a2d8bab5a222015503deeba340e02451a46cf2a7f0c50b3e52a305c76f"}
Oct 30 22:05:22.135376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:22.135320    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podStartSLOduration=2.13530358 podStartE2EDuration="2.13530358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:22.133071563 +0000 UTC m=+933.657214339" watchObservedRunningTime="2026-10-30 22:05:22.13530358 +0000 UTC m=+933.659446343"
Oct 30 22:05:22.289297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:22.289033    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:22.291151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:22.291122    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:22.291314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:22.291172    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:23.040916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:23.040882    5031 generic.go:356] "Generic (PLEG): container finished" podID="7b048e45-b26f-4a64-a26c-4d6eea108f2f" containerID="1ada363e8bdd980ce5990ea2db18a17700e3515f8c15dfb28c818f19bd258c7b" exitCode=0
Oct 30 22:05:23.041109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:23.040964    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl" event={"ID":"7b048e45-b26f-4a64-a26c-4d6eea108f2f","Type":"ContainerDied","Data":"1ada363e8bdd980ce5990ea2db18a17700e3515f8c15dfb28c818f19bd258c7b"}
Oct 30 22:05:23.288900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:23.288812    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:23.289045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:23.288913    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:24.288410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.288382    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:24.288720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.288436    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:24.299588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.299530    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:24.466762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.466724    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-openshift-ca\") pod \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") "
Oct 30 22:05:24.466942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.466894    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-xks-ca\") pod \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") "
Oct 30 22:05:24.467311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.467097    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b048e45-b26f-4a64-a26c-4d6eea108f2f-kube-api-access-nx4n8\" (UniqueName: \"kubernetes.io/projected/7b048e45-b26f-4a64-a26c-4d6eea108f2f-kube-api-access-nx4n8\") pod \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\" (UID: \"7b048e45-b26f-4a64-a26c-4d6eea108f2f\") "
Oct 30 22:05:24.467718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.467699    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-openshift-ca" pod "7b048e45-b26f-4a64-a26c-4d6eea108f2f" (UID: "7b048e45-b26f-4a64-a26c-4d6eea108f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:24.468196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.468174    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:24.469614 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.469583    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-xks-ca" pod "7b048e45-b26f-4a64-a26c-4d6eea108f2f" (UID: "7b048e45-b26f-4a64-a26c-4d6eea108f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:24.470904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.470881    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b048e45-b26f-4a64-a26c-4d6eea108f2f-kube-api-access-nx4n8" pod "7b048e45-b26f-4a64-a26c-4d6eea108f2f" (UID: "7b048e45-b26f-4a64-a26c-4d6eea108f2f"). InnerVolumeSpecName "kube-api-access-nx4n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:24.571054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.570969    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b048e45-b26f-4a64-a26c-4d6eea108f2f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:24.571271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.571257    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx4n8\" (UniqueName: \"kubernetes.io/projected/7b048e45-b26f-4a64-a26c-4d6eea108f2f-kube-api-access-nx4n8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:24.869307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:24.869272    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:25.060142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:25.060112    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"
Oct 30 22:05:25.060362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:25.060142    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl" event={"ID":"7b048e45-b26f-4a64-a26c-4d6eea108f2f","Type":"ContainerDied","Data":"638cb15c3e0ae930f79203489dd21d4c3d8f544c83cfa773e6579ec0d6dd6e14"}
Oct 30 22:05:25.060362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:25.060255    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638cb15c3e0ae930f79203489dd21d4c3d8f544c83cfa773e6579ec0d6dd6e14"
Oct 30 22:05:25.288750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:25.288665    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:25.288750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:25.288734    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:26.269807 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.269772    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:26.288819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.288787    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:26.289257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.288843    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:26.454723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.454687    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"]
Oct 30 22:05:26.455788 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.455756    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7b048e45-b26f-4a64-a26c-4d6eea108f2f" containerName="revoke-keys"
Oct 30 22:05:26.455935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.455925    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b048e45-b26f-4a64-a26c-4d6eea108f2f" containerName="revoke-keys"
Oct 30 22:05:26.456148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.456137    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b048e45-b26f-4a64-a26c-4d6eea108f2f" containerName="revoke-keys"
Oct 30 22:05:26.470498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.470472    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.537284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.534870    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"]
Oct 30 22:05:26.572817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.572418    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:26.572817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.572574    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:26.572817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.572610    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:26.572817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.572691    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:26.600813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.600780    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.600813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.600815    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kslbg\" (UniqueName: \"kubernetes.io/projected/a3d4178c-eafe-4823-9772-85691ae354e7-kube-api-access-kslbg\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.601020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.600935    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.701756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.701724    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.701756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.701759    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kslbg\" (UniqueName: \"kubernetes.io/projected/a3d4178c-eafe-4823-9772-85691ae354e7-kube-api-access-kslbg\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.702030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.701830    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.702519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.702487    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.704624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.704605    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.767255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.767224    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kslbg\" (UniqueName: \"kubernetes.io/projected/a3d4178c-eafe-4823-9772-85691ae354e7-kube-api-access-kslbg\") pod \"maas-api-revoke-sub-7df5c6f4-84hfb\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:26.793230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:26.793141    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:27.233533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.233475    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"]
Oct 30 22:05:27.246708 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:27.246676    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d4178c_eafe_4823_9772_85691ae354e7.slice/crio-f3730dacc2a6e4d48c801f66d198ef4c16cc9f8216c7126b3affce9a14ff5503 WatchSource:0}: Error finding container f3730dacc2a6e4d48c801f66d198ef4c16cc9f8216c7126b3affce9a14ff5503: Status 404 returned error can't find the container with id f3730dacc2a6e4d48c801f66d198ef4c16cc9f8216c7126b3affce9a14ff5503
Oct 30 22:05:27.289398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.289364    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:27.289710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.289430    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:27.648508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.648473    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"]
Oct 30 22:05:27.654064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.654042    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.729248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.728152    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c69c\" (UniqueName: \"kubernetes.io/projected/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-kube-api-access-9c69c\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.729248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.728224    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.729248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.728312    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.742106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.740798    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"]
Oct 30 22:05:27.830287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.829741    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.830287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.829942    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c69c\" (UniqueName: \"kubernetes.io/projected/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-kube-api-access-9c69c\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.830287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.829981    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.831542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.830828    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.835528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.835502    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.894939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.894903    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c69c\" (UniqueName: \"kubernetes.io/projected/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-kube-api-access-9c69c\") pod \"maas-api-revoke-sub-d356c69e-nh86l\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:27.969030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:27.968999    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:28.095055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.095015    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb" event={"ID":"a3d4178c-eafe-4823-9772-85691ae354e7","Type":"ContainerStarted","Data":"ceb42943c6075a7927bcb9bbd39c554219bad108267da8025dabf7674762a235"}
Oct 30 22:05:28.095380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.095081    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb" event={"ID":"a3d4178c-eafe-4823-9772-85691ae354e7","Type":"ContainerStarted","Data":"f3730dacc2a6e4d48c801f66d198ef4c16cc9f8216c7126b3affce9a14ff5503"}
Oct 30 22:05:28.216746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.216691    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb" podStartSLOduration=2.216675418 podStartE2EDuration="2.216675418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:28.214744414 +0000 UTC m=+939.738887180" watchObservedRunningTime="2026-10-30 22:05:28.216675418 +0000 UTC m=+939.740818191"
Oct 30 22:05:28.289220 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.289173    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:28.289373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.289245    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:28.381054 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:28.381005    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8947ad21_f13a_4ecc_bdae_9bf35fe4a2c8.slice/crio-0203cc9a87e59cbd7d7eced668df3f31858747399eb16a9a0ca282ba13abee96 WatchSource:0}: Error finding container 0203cc9a87e59cbd7d7eced668df3f31858747399eb16a9a0ca282ba13abee96: Status 404 returned error can't find the container with id 0203cc9a87e59cbd7d7eced668df3f31858747399eb16a9a0ca282ba13abee96
Oct 30 22:05:28.381553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.381524    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"]
Oct 30 22:05:28.526468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.526431    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"]
Oct 30 22:05:28.533016 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.532993    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.546799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.546763    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzlv\" (UniqueName: \"kubernetes.io/projected/0d697ea1-6c47-4633-b963-07b81688b100-kube-api-access-mdzlv\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.546948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.546835    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.547081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.547053    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.590656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.590621    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"]
Oct 30 22:05:28.647830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.647788    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.648019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.647949    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzlv\" (UniqueName: \"kubernetes.io/projected/0d697ea1-6c47-4633-b963-07b81688b100-kube-api-access-mdzlv\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.648091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.648052    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.648758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.648737    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.650432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.650416    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.712188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.712155    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzlv\" (UniqueName: \"kubernetes.io/projected/0d697ea1-6c47-4633-b963-07b81688b100-kube-api-access-mdzlv\") pod \"maas-api-revoke-sub-40e60243-dxmqf\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:28.862018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:28.861983    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:29.107309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.107240    5031 generic.go:356] "Generic (PLEG): container finished" podID="a3d4178c-eafe-4823-9772-85691ae354e7" containerID="ceb42943c6075a7927bcb9bbd39c554219bad108267da8025dabf7674762a235" exitCode=0
Oct 30 22:05:29.107309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.107309    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb" event={"ID":"a3d4178c-eafe-4823-9772-85691ae354e7","Type":"ContainerDied","Data":"ceb42943c6075a7927bcb9bbd39c554219bad108267da8025dabf7674762a235"}
Oct 30 22:05:29.109858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.109813    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l" event={"ID":"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8","Type":"ContainerStarted","Data":"53caea505561cc56d364666c95c8c08e258c18465a27c436326473e4896a2af7"}
Oct 30 22:05:29.109858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.109863    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l" event={"ID":"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8","Type":"ContainerStarted","Data":"0203cc9a87e59cbd7d7eced668df3f31858747399eb16a9a0ca282ba13abee96"}
Oct 30 22:05:29.286713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.286682    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"]
Oct 30 22:05:29.288399 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:29.288363    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d697ea1_6c47_4633_b963_07b81688b100.slice/crio-75f379df116081ab901ae2255959602f7730e6f7b0e761745b9c9e27d341eac7 WatchSource:0}: Error finding container 75f379df116081ab901ae2255959602f7730e6f7b0e761745b9c9e27d341eac7: Status 404 returned error can't find the container with id 75f379df116081ab901ae2255959602f7730e6f7b0e761745b9c9e27d341eac7
Oct 30 22:05:29.288531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.288479    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:05:29.288591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.288530    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 22:05:29.361780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.361697    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l" podStartSLOduration=2.361673534 podStartE2EDuration="2.361673534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:29.356929784 +0000 UTC m=+940.881072575" watchObservedRunningTime="2026-10-30 22:05:29.361673534 +0000 UTC m=+940.885816298"
Oct 30 22:05:29.960296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:29.955156    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:30.141234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.140613    5031 generic.go:356] "Generic (PLEG): container finished" podID="8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" containerID="53caea505561cc56d364666c95c8c08e258c18465a27c436326473e4896a2af7" exitCode=0
Oct 30 22:05:30.141234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.140702    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l" event={"ID":"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8","Type":"ContainerDied","Data":"53caea505561cc56d364666c95c8c08e258c18465a27c436326473e4896a2af7"}
Oct 30 22:05:30.162558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.146879    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf" event={"ID":"0d697ea1-6c47-4633-b963-07b81688b100","Type":"ContainerStarted","Data":"6111937e7902d78902ef965bf8a339ff0a888a83ce3b390e358518131ea55c47"}
Oct 30 22:05:30.162558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.146956    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf" event={"ID":"0d697ea1-6c47-4633-b963-07b81688b100","Type":"ContainerStarted","Data":"75f379df116081ab901ae2255959602f7730e6f7b0e761745b9c9e27d341eac7"}
Oct 30 22:05:30.501920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.488960    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf" podStartSLOduration=2.488940468 podStartE2EDuration="2.488940468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:30.38427634 +0000 UTC m=+941.908419123" watchObservedRunningTime="2026-10-30 22:05:30.488940468 +0000 UTC m=+942.013083233"
Oct 30 22:05:30.660114 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.660086    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:30.703300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.703266    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d4178c-eafe-4823-9772-85691ae354e7-kube-api-access-kslbg\" (UniqueName: \"kubernetes.io/projected/a3d4178c-eafe-4823-9772-85691ae354e7-kube-api-access-kslbg\") pod \"a3d4178c-eafe-4823-9772-85691ae354e7\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") "
Oct 30 22:05:30.703514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.703392    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-openshift-ca\") pod \"a3d4178c-eafe-4823-9772-85691ae354e7\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") "
Oct 30 22:05:30.703514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.703431    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-xks-ca\") pod \"a3d4178c-eafe-4823-9772-85691ae354e7\" (UID: \"a3d4178c-eafe-4823-9772-85691ae354e7\") "
Oct 30 22:05:30.703876 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.703842    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-openshift-ca" pod "a3d4178c-eafe-4823-9772-85691ae354e7" (UID: "a3d4178c-eafe-4823-9772-85691ae354e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:30.705851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.705819    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-xks-ca" pod "a3d4178c-eafe-4823-9772-85691ae354e7" (UID: "a3d4178c-eafe-4823-9772-85691ae354e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:30.705987 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.705862    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d4178c-eafe-4823-9772-85691ae354e7-kube-api-access-kslbg" pod "a3d4178c-eafe-4823-9772-85691ae354e7" (UID: "a3d4178c-eafe-4823-9772-85691ae354e7"). InnerVolumeSpecName "kube-api-access-kslbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:30.805631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.805594    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.805631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.805626    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d4178c-eafe-4823-9772-85691ae354e7-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.805631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.805639    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kslbg\" (UniqueName: \"kubernetes.io/projected/a3d4178c-eafe-4823-9772-85691ae354e7-kube-api-access-kslbg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.837760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.837675    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"]
Oct 30 22:05:30.838423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.838401    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a3d4178c-eafe-4823-9772-85691ae354e7" containerName="revoke-keys"
Oct 30 22:05:30.838423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.838423    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d4178c-eafe-4823-9772-85691ae354e7" containerName="revoke-keys"
Oct 30 22:05:30.838632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.838617    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d4178c-eafe-4823-9772-85691ae354e7" containerName="revoke-keys"
Oct 30 22:05:30.842479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.842458    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:30.912627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.909282    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"]
Oct 30 22:05:30.912627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.909777    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8ca52a-openshift-default-dockercfg-fdgr2\""
Oct 30 22:05:30.949588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.949547    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:30.950860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.950808    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:30.951604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:30.951557    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:05:31.013637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.013590    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.013829    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkj2h\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-kube-api-access-fkj2h\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.014164    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.014269    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.014314    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.014369    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.014455    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.014508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5919da72-ea5a-46a5-8fbd-32e6266099de-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.014736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.014583    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.115595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.115555    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.115767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.115638    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkj2h\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-kube-api-access-fkj2h\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.115767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.115678    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.115903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.115862    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.115958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.115909    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.116053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.116029    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.116223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.116087    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.116223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.116114    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5919da72-ea5a-46a5-8fbd-32e6266099de-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.116223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.116137    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.116223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.116145    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.116449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.116233    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.116956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.116929    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.117066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.117033    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.117246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.117201    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5919da72-ea5a-46a5-8fbd-32e6266099de-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.118897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.118869    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.119104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.119082    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.155709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.154980    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"
Oct 30 22:05:31.155709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.154974    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb" event={"ID":"a3d4178c-eafe-4823-9772-85691ae354e7","Type":"ContainerDied","Data":"f3730dacc2a6e4d48c801f66d198ef4c16cc9f8216c7126b3affce9a14ff5503"}
Oct 30 22:05:31.155709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.155061    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3730dacc2a6e4d48c801f66d198ef4c16cc9f8216c7126b3affce9a14ff5503"
Oct 30 22:05:31.160665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.157156    5031 generic.go:356] "Generic (PLEG): container finished" podID="0d697ea1-6c47-4633-b963-07b81688b100" containerID="6111937e7902d78902ef965bf8a339ff0a888a83ce3b390e358518131ea55c47" exitCode=0
Oct 30 22:05:31.160665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.158178    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf" event={"ID":"0d697ea1-6c47-4633-b963-07b81688b100","Type":"ContainerDied","Data":"6111937e7902d78902ef965bf8a339ff0a888a83ce3b390e358518131ea55c47"}
Oct 30 22:05:31.179822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.179793    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.179988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.179931    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkj2h\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-kube-api-access-fkj2h\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.334342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.334313    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:31.419385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.419292    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-kube-api-access-9c69c\" (UniqueName: \"kubernetes.io/projected/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-kube-api-access-9c69c\") pod \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") "
Oct 30 22:05:31.419556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.419445    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-openshift-ca\") pod \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") "
Oct 30 22:05:31.419556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.419495    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-xks-ca\") pod \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\" (UID: \"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8\") "
Oct 30 22:05:31.419843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.419817    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-openshift-ca" pod "8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" (UID: "8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:31.420064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.420049    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:31.421746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.421717    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-xks-ca" pod "8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" (UID: "8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:31.421867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.421768    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-kube-api-access-9c69c" pod "8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" (UID: "8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8"). InnerVolumeSpecName "kube-api-access-9c69c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:31.458887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.458855    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:31.521515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.521486    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c69c\" (UniqueName: \"kubernetes.io/projected/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-kube-api-access-9c69c\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:31.521690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.521679    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:31.916967 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:31.916938    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5919da72_ea5a_46a5_8fbd_32e6266099de.slice/crio-fc2df62967a12eda5e38010bfeaf9a2b276f8f43951ecb9b1a1dfe6db67df87b WatchSource:0}: Error finding container fc2df62967a12eda5e38010bfeaf9a2b276f8f43951ecb9b1a1dfe6db67df87b: Status 404 returned error can't find the container with id fc2df62967a12eda5e38010bfeaf9a2b276f8f43951ecb9b1a1dfe6db67df87b
Oct 30 22:05:31.917710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.917683    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"]
Oct 30 22:05:31.919937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.919904    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:31.920033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.919968    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:31.920033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:31.920001    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:32.167002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.166965    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"
Oct 30 22:05:32.167449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.166965    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l" event={"ID":"8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8","Type":"ContainerDied","Data":"0203cc9a87e59cbd7d7eced668df3f31858747399eb16a9a0ca282ba13abee96"}
Oct 30 22:05:32.167449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.167099    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0203cc9a87e59cbd7d7eced668df3f31858747399eb16a9a0ca282ba13abee96"
Oct 30 22:05:32.169055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.169026    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" event={"ID":"5919da72-ea5a-46a5-8fbd-32e6266099de","Type":"ContainerStarted","Data":"f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160"}
Oct 30 22:05:32.169188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.169079    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" event={"ID":"5919da72-ea5a-46a5-8fbd-32e6266099de","Type":"ContainerStarted","Data":"fc2df62967a12eda5e38010bfeaf9a2b276f8f43951ecb9b1a1dfe6db67df87b"}
Oct 30 22:05:32.306443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.306374    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podStartSLOduration=2.306353873 podStartE2EDuration="2.306353873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:32.299599719 +0000 UTC m=+943.823742483" watchObservedRunningTime="2026-10-30 22:05:32.306353873 +0000 UTC m=+943.830496638"
Oct 30 22:05:32.370532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.370502    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:32.434508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.432816    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-xks-ca\") pod \"0d697ea1-6c47-4633-b963-07b81688b100\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") "
Oct 30 22:05:32.434508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.432927    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d697ea1-6c47-4633-b963-07b81688b100-kube-api-access-mdzlv\" (UniqueName: \"kubernetes.io/projected/0d697ea1-6c47-4633-b963-07b81688b100-kube-api-access-mdzlv\") pod \"0d697ea1-6c47-4633-b963-07b81688b100\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") "
Oct 30 22:05:32.434508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.433100    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-openshift-ca\") pod \"0d697ea1-6c47-4633-b963-07b81688b100\" (UID: \"0d697ea1-6c47-4633-b963-07b81688b100\") "
Oct 30 22:05:32.434508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.433614    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-openshift-ca" pod "0d697ea1-6c47-4633-b963-07b81688b100" (UID: "0d697ea1-6c47-4633-b963-07b81688b100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:32.434508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.433922    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:32.435934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.435902    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-xks-ca" pod "0d697ea1-6c47-4633-b963-07b81688b100" (UID: "0d697ea1-6c47-4633-b963-07b81688b100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:32.436458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.436437    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d697ea1-6c47-4633-b963-07b81688b100-kube-api-access-mdzlv" pod "0d697ea1-6c47-4633-b963-07b81688b100" (UID: "0d697ea1-6c47-4633-b963-07b81688b100"). InnerVolumeSpecName "kube-api-access-mdzlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:32.460193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.460169    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:32.535044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.534949    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d697ea1-6c47-4633-b963-07b81688b100-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:32.535044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:32.534989    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdzlv\" (UniqueName: \"kubernetes.io/projected/0d697ea1-6c47-4633-b963-07b81688b100-kube-api-access-mdzlv\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:33.178192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:33.178159    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"
Oct 30 22:05:33.178630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:33.178198    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf" event={"ID":"0d697ea1-6c47-4633-b963-07b81688b100","Type":"ContainerDied","Data":"75f379df116081ab901ae2255959602f7730e6f7b0e761745b9c9e27d341eac7"}
Oct 30 22:05:33.178630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:33.178259    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f379df116081ab901ae2255959602f7730e6f7b0e761745b9c9e27d341eac7"
Oct 30 22:05:33.462163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:33.462076    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:05:33.462163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:33.462131    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:05:34.460373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:34.460331    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:05:34.460811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:34.460390    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:05:35.020049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:35.011812    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:35.460328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:35.460296    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:05:35.460548 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:35.460354    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:05:36.459591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:36.459557    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:05:36.459792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:36.459618    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:05:37.460434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:37.460399    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:05:37.460826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:37.460462    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:05:38.459766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:38.459729    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:05:38.459937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:38.459811    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:05:39.341390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.341359    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"]
Oct 30 22:05:39.342037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.342023    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" containerName="revoke-keys"
Oct 30 22:05:39.342076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.342039    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" containerName="revoke-keys"
Oct 30 22:05:39.342076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.342053    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0d697ea1-6c47-4633-b963-07b81688b100" containerName="revoke-keys"
Oct 30 22:05:39.342076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.342057    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d697ea1-6c47-4633-b963-07b81688b100" containerName="revoke-keys"
Oct 30 22:05:39.342183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.342153    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d697ea1-6c47-4633-b963-07b81688b100" containerName="revoke-keys"
Oct 30 22:05:39.342183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.342165    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" containerName="revoke-keys"
Oct 30 22:05:39.346728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.346709    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.408840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.408804    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"]
Oct 30 22:05:39.454285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.454252    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:39.454285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.454253    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:39.454285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.454299    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:39.454565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.454400    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:39.454773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.454750    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.454816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.454791    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58h6\" (UniqueName: \"kubernetes.io/projected/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-kube-api-access-k58h6\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.454865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.454819    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.464918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.464891    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:39.465194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.465174    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:39.465977 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.465955    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:05:39.556057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.556025    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58h6\" (UniqueName: \"kubernetes.io/projected/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-kube-api-access-k58h6\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.556280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.556074    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.556480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.556454    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.556853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.556833    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.559010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.558986    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.620110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.620080    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58h6\" (UniqueName: \"kubernetes.io/projected/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-kube-api-access-k58h6\") pod \"maas-api-revoke-sub-9cc4c407-t6jgj\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:39.661380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:39.661344    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:40.018121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:40.018018    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:40.049555 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:40.049522    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d033d7_48eb_4412_9a0d_5b9bbe392ccb.slice/crio-58c62a7049f8cff397d06cfd2a63c4856edeb60ed0d7d1274cecb49d27f589ec WatchSource:0}: Error finding container 58c62a7049f8cff397d06cfd2a63c4856edeb60ed0d7d1274cecb49d27f589ec: Status 404 returned error can't find the container with id 58c62a7049f8cff397d06cfd2a63c4856edeb60ed0d7d1274cecb49d27f589ec
Oct 30 22:05:40.049974 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:40.049949    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"]
Oct 30 22:05:40.262590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:40.262550    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj" event={"ID":"77d033d7-48eb-4412-9a0d-5b9bbe392ccb","Type":"ContainerStarted","Data":"58c62a7049f8cff397d06cfd2a63c4856edeb60ed0d7d1274cecb49d27f589ec"}
Oct 30 22:05:41.271480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:41.271376    5031 generic.go:356] "Generic (PLEG): container finished" podID="77d033d7-48eb-4412-9a0d-5b9bbe392ccb" containerID="9b13f510e650ab6bd7efd1fad647e9b9fd0795381157d8710d1a8078ffb5878b" exitCode=0
Oct 30 22:05:41.271480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:41.271463    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj" event={"ID":"77d033d7-48eb-4412-9a0d-5b9bbe392ccb","Type":"ContainerDied","Data":"9b13f510e650ab6bd7efd1fad647e9b9fd0795381157d8710d1a8078ffb5878b"}
Oct 30 22:05:42.467023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.466987    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:42.592751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.592722    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-openshift-ca\") pod \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") "
Oct 30 22:05:42.592916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.592832    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-kube-api-access-k58h6\" (UniqueName: \"kubernetes.io/projected/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-kube-api-access-k58h6\") pod \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") "
Oct 30 22:05:42.592991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.592973    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-xks-ca\") pod \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\" (UID: \"77d033d7-48eb-4412-9a0d-5b9bbe392ccb\") "
Oct 30 22:05:42.593087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.593060    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-openshift-ca" pod "77d033d7-48eb-4412-9a0d-5b9bbe392ccb" (UID: "77d033d7-48eb-4412-9a0d-5b9bbe392ccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:42.593463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.593443    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:42.595180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.595157    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-kube-api-access-k58h6" pod "77d033d7-48eb-4412-9a0d-5b9bbe392ccb" (UID: "77d033d7-48eb-4412-9a0d-5b9bbe392ccb"). InnerVolumeSpecName "kube-api-access-k58h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:42.595826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.595801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-xks-ca" pod "77d033d7-48eb-4412-9a0d-5b9bbe392ccb" (UID: "77d033d7-48eb-4412-9a0d-5b9bbe392ccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:42.694431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.694403    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k58h6\" (UniqueName: \"kubernetes.io/projected/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-kube-api-access-k58h6\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:42.694431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:42.694430    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d033d7-48eb-4412-9a0d-5b9bbe392ccb-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:43.296508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:43.296475    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"
Oct 30 22:05:43.296508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:43.296498    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj" event={"ID":"77d033d7-48eb-4412-9a0d-5b9bbe392ccb","Type":"ContainerDied","Data":"58c62a7049f8cff397d06cfd2a63c4856edeb60ed0d7d1274cecb49d27f589ec"}
Oct 30 22:05:43.296729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:43.296577    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c62a7049f8cff397d06cfd2a63c4856edeb60ed0d7d1274cecb49d27f589ec"
Oct 30 22:05:44.040266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:44.040197    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:44.321707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:44.321616    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:45.099110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.099073    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:45.327660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.327622    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"]
Oct 30 22:05:45.328316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.328301    5031 cpu_manager.go:414] "containerMap: removing container" podUID="77d033d7-48eb-4412-9a0d-5b9bbe392ccb" containerName="revoke-keys"
Oct 30 22:05:45.328316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.328317    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d033d7-48eb-4412-9a0d-5b9bbe392ccb" containerName="revoke-keys"
Oct 30 22:05:45.328430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.328412    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d033d7-48eb-4412-9a0d-5b9bbe392ccb" containerName="revoke-keys"
Oct 30 22:05:45.332280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.332264    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.428824    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429053    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a217c30-ee84-4544-9ff6-d03466b88f7b-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429118    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429184    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429240    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429278    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmhn\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-kube-api-access-brmhn\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429350    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429422    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429441    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"]
Oct 30 22:05:45.431047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.429483    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531024    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmhn\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-kube-api-access-brmhn\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531104    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531168    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531262    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531338    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531383    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a217c30-ee84-4544-9ff6-d03466b88f7b-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531449    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531516    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531555    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531695    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.531843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.531766    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.532083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.532056    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.532230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.532188    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.532392 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.532372    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a217c30-ee84-4544-9ff6-d03466b88f7b-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.534380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.534360    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.534506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.534490    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.594773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.594735    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.594923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.594878    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmhn\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-kube-api-access-brmhn\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:45.648622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:45.648582    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:46.074274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.074169    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:05:46.092062 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:46.092029    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a217c30_ee84_4544_9ff6_d03466b88f7b.slice/crio-ee425a77d6c6b43674508858d8f95486eda6fa22b5b055918e28b0a69f77749a WatchSource:0}: Error finding container ee425a77d6c6b43674508858d8f95486eda6fa22b5b055918e28b0a69f77749a: Status 404 returned error can't find the container with id ee425a77d6c6b43674508858d8f95486eda6fa22b5b055918e28b0a69f77749a
Oct 30 22:05:46.093124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.093102    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"]
Oct 30 22:05:46.095003 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.094968    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:46.095066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.095050    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:46.095104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.095091    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:05:46.333858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.333706    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" event={"ID":"0a217c30-ee84-4544-9ff6-d03466b88f7b","Type":"ContainerStarted","Data":"ed7e3058036a129f410de6a728c5cd8326cb52af5852074c27843d0c95f57f13"}
Oct 30 22:05:46.333858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.333768    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" event={"ID":"0a217c30-ee84-4544-9ff6-d03466b88f7b","Type":"ContainerStarted","Data":"ee425a77d6c6b43674508858d8f95486eda6fa22b5b055918e28b0a69f77749a"}
Oct 30 22:05:46.679269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:46.678461    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:47.663697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:47.663654    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:05:47.664393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:47.664359    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 22:05:48.649948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:48.649912    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:05:48.650179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:48.649986    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 22:05:49.649170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:49.649129    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:05:49.649658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:49.649268    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 22:05:50.334233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:50.334126    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:50.560228 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:05:50.560165    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:05:50.649612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:50.649579    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:05:50.650036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:50.649628    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 22:05:51.649806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:51.649770    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:05:51.650241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:51.649823    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 22:05:52.649028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:52.648991    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:05:52.649236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:52.649046    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 22:05:53.913274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:53.913242    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:53.913823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:53.913647    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:53.914528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:53.914506    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:05:54.093047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:54.092978    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podStartSLOduration=9.092959133 podStartE2EDuration="9.092959133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:46.492069214 +0000 UTC m=+958.016211977" watchObservedRunningTime="2026-10-30 22:05:54.092959133 +0000 UTC m=+965.617101902"
Oct 30 22:05:55.196119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.196085    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"]
Oct 30 22:05:55.201945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.201923    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.258751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.258718    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"]
Oct 30 22:05:55.304006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.303974    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:55.304006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.303974    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:55.304283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.304031    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:05:55.306179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.306152    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:55.326064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.326036    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92r92\" (UniqueName: \"kubernetes.io/projected/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-kube-api-access-92r92\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.326247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.326182    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.326343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.326291    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.405982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.405943    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:55.427994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.427958    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.428180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.428040    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.428274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.428182    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92r92\" (UniqueName: \"kubernetes.io/projected/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-kube-api-access-92r92\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.428669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.428649    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.430607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.430586    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.492702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.492628    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92r92\" (UniqueName: \"kubernetes.io/projected/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-kube-api-access-92r92\") pod \"maas-api-revoke-sub-05ffe514-zxbk2\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.517781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.517750    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:55.900340 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:05:55.900307    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3017489f_4cbb_4f8a_b5a8_4b954e78db1b.slice/crio-24688b717d4c81f3a866ffbb7bd5b5bdca5b13f4ff2db6c3a98fe1a8cc6b4564 WatchSource:0}: Error finding container 24688b717d4c81f3a866ffbb7bd5b5bdca5b13f4ff2db6c3a98fe1a8cc6b4564: Status 404 returned error can't find the container with id 24688b717d4c81f3a866ffbb7bd5b5bdca5b13f4ff2db6c3a98fe1a8cc6b4564
Oct 30 22:05:55.901378 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:55.901351    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"]
Oct 30 22:05:56.492829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:56.492732    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2" event={"ID":"3017489f-4cbb-4f8a-b5a8-4b954e78db1b","Type":"ContainerStarted","Data":"c81515b24a8770baf781ba1b71b4a42c402d5b3afc7800d67e193d5e9cb6ae2f"}
Oct 30 22:05:56.492829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:56.492789    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2" event={"ID":"3017489f-4cbb-4f8a-b5a8-4b954e78db1b","Type":"ContainerStarted","Data":"24688b717d4c81f3a866ffbb7bd5b5bdca5b13f4ff2db6c3a98fe1a8cc6b4564"}
Oct 30 22:05:56.614011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:56.613956    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2" podStartSLOduration=1.613938568 podStartE2EDuration="1.613938568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:56.612424115 +0000 UTC m=+968.136566904" watchObservedRunningTime="2026-10-30 22:05:56.613938568 +0000 UTC m=+968.138081332"
Oct 30 22:05:57.501997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:57.501958    5031 generic.go:356] "Generic (PLEG): container finished" podID="3017489f-4cbb-4f8a-b5a8-4b954e78db1b" containerID="c81515b24a8770baf781ba1b71b4a42c402d5b3afc7800d67e193d5e9cb6ae2f" exitCode=0
Oct 30 22:05:57.502445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:57.502013    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2" event={"ID":"3017489f-4cbb-4f8a-b5a8-4b954e78db1b","Type":"ContainerDied","Data":"c81515b24a8770baf781ba1b71b4a42c402d5b3afc7800d67e193d5e9cb6ae2f"}
Oct 30 22:05:58.665123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.665089    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:58.770400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.770318    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-kube-api-access-92r92\" (UniqueName: \"kubernetes.io/projected/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-kube-api-access-92r92\") pod \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") "
Oct 30 22:05:58.770577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.770412    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-openshift-ca\") pod \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") "
Oct 30 22:05:58.770577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.770443    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-xks-ca\") pod \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\" (UID: \"3017489f-4cbb-4f8a-b5a8-4b954e78db1b\") "
Oct 30 22:05:58.770899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.770869    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-openshift-ca" pod "3017489f-4cbb-4f8a-b5a8-4b954e78db1b" (UID: "3017489f-4cbb-4f8a-b5a8-4b954e78db1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:58.771140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.771118    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:58.772822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.772796    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-kube-api-access-92r92" pod "3017489f-4cbb-4f8a-b5a8-4b954e78db1b" (UID: "3017489f-4cbb-4f8a-b5a8-4b954e78db1b"). InnerVolumeSpecName "kube-api-access-92r92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:58.773046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.773025    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-xks-ca" pod "3017489f-4cbb-4f8a-b5a8-4b954e78db1b" (UID: "3017489f-4cbb-4f8a-b5a8-4b954e78db1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:58.872298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.872264    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92r92\" (UniqueName: \"kubernetes.io/projected/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-kube-api-access-92r92\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:58.872298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:58.872292    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3017489f-4cbb-4f8a-b5a8-4b954e78db1b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:59.525316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:59.525271    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2" event={"ID":"3017489f-4cbb-4f8a-b5a8-4b954e78db1b","Type":"ContainerDied","Data":"24688b717d4c81f3a866ffbb7bd5b5bdca5b13f4ff2db6c3a98fe1a8cc6b4564"}
Oct 30 22:05:59.525316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:59.525308    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"
Oct 30 22:05:59.525550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:05:59.525359    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24688b717d4c81f3a866ffbb7bd5b5bdca5b13f4ff2db6c3a98fe1a8cc6b4564"
Oct 30 22:06:00.423313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.423266    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:00.793627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.793547    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"]
Oct 30 22:06:00.794182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.794168    5031 cpu_manager.go:414] "containerMap: removing container" podUID="3017489f-4cbb-4f8a-b5a8-4b954e78db1b" containerName="revoke-keys"
Oct 30 22:06:00.794251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.794183    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3017489f-4cbb-4f8a-b5a8-4b954e78db1b" containerName="revoke-keys"
Oct 30 22:06:00.794307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.794297    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="3017489f-4cbb-4f8a-b5a8-4b954e78db1b" containerName="revoke-keys"
Oct 30 22:06:00.799072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.799052    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.880285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.880256    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"]
Oct 30 22:06:00.898745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.898711    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.898927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.898751    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmkz\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-kube-api-access-4nmkz\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.898927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.898782    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.898927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.898813    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.898927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.898868    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.898927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.898896    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541b324-ba5a-41d5-b979-9ba0d860117a-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.899178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.898946    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.899178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.899031    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:00.899178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:00.899126    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.000907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.000860    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.000978    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001008    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmkz\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-kube-api-access-4nmkz\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001039    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001162    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001280    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001316    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541b324-ba5a-41d5-b979-9ba0d860117a-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001370    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001442    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001450    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001524    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001712    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.001797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.001780    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.002123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.002060    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541b324-ba5a-41d5-b979-9ba0d860117a-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.003669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.003645    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.003993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.003975    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.064633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.064573    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.065089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.065069    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmkz\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-kube-api-access-4nmkz\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.113682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.113652    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:01.560503 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:06:01.560474    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1541b324_ba5a_41d5_b979_9ba0d860117a.slice/crio-b16692f9c606a79622413301162b40e4674a93a7a71003acb64ed49c7406ee09 WatchSource:0}: Error finding container b16692f9c606a79622413301162b40e4674a93a7a71003acb64ed49c7406ee09: Status 404 returned error can't find the container with id b16692f9c606a79622413301162b40e4674a93a7a71003acb64ed49c7406ee09
Oct 30 22:06:01.561074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.561048    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"]
Oct 30 22:06:01.563238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.563171    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:01.563330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.563266    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:01.563330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:01.563299    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:02.551204 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:02.551163    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" event={"ID":"1541b324-ba5a-41d5-b979-9ba0d860117a","Type":"ContainerStarted","Data":"ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e"}
Oct 30 22:06:02.551419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:02.551260    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" event={"ID":"1541b324-ba5a-41d5-b979-9ba0d860117a","Type":"ContainerStarted","Data":"b16692f9c606a79622413301162b40e4674a93a7a71003acb64ed49c7406ee09"}
Oct 30 22:06:02.678482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:02.678429    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podStartSLOduration=2.678411326 podStartE2EDuration="2.678411326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:02.674664225 +0000 UTC m=+974.198807007" watchObservedRunningTime="2026-10-30 22:06:02.678411326 +0000 UTC m=+974.202554091"
Oct 30 22:06:03.118234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:03.116832    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:03.118234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:03.117136    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:06:03.118234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:03.117260    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:06:04.115624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:04.115584    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:06:04.116337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:04.116291    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:06:05.129236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:05.116638    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:06:05.129236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:05.116714    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:06:05.786749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:05.786693    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:06.119235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:06.118593    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:06:06.119235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:06.118669    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:06:07.129236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:07.128601    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:06:07.129236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:07.128663    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:06:08.254837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:08.254798    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:08.255368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:08.255166    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:08.256101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:08.256082    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:06:10.915750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:10.915715    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:15.999508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:15.999471    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:19.211120 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:19.211078    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:06:20.550039 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:06:20.550003    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:06:20.668862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:20.668825    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:21.165295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:21.165259    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:26.105064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.105029    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:26.211666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.211628    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"]
Oct 30 22:06:26.217269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.217248    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.252166 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.252130    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-model-cache\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.252385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.252360    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-tmp-dir\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.252451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.252396    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-dshm\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.252451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.252422    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-home\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.252568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.252469    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f65885-52f6-4462-90cf-dd806344c29a-tls-certs\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.252568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.252508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/a9f65885-52f6-4462-90cf-dd806344c29a-kube-api-access-7qhf2\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.274465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.274430    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"]
Oct 30 22:06:26.318757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.318715    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"openshift-service-ca.crt\""
Oct 30 22:06:26.318757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.318751    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"distinct-w5-3783c2-kserve-self-signed-certs\""
Oct 30 22:06:26.321706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.321681    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"default-dockercfg-f8swn\""
Oct 30 22:06:26.321706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.321690    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"kube-root-ca.crt\""
Oct 30 22:06:26.353508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.353473    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/a9f65885-52f6-4462-90cf-dd806344c29a-kube-api-access-7qhf2\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.353731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.353541    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-model-cache\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.353731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.353685    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-tmp-dir\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.353731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.353707    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-dshm\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.353731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.353725    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-home\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.353954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.353777    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f65885-52f6-4462-90cf-dd806344c29a-tls-certs\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.354011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.353986    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-model-cache\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.354083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.354061    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-tmp-dir\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.354273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.354256    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-home\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.356598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.356527    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-dshm\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.375448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.375412    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f65885-52f6-4462-90cf-dd806344c29a-tls-certs\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.415721 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.415657    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:26.417956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.417933    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/a9f65885-52f6-4462-90cf-dd806344c29a-kube-api-access-7qhf2\") pod \"distinct-w5-3783c2-kserve-79dd8c88b6-lkx27\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") " pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.533001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.532963    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:26.969903 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:06:26.969867    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9f65885_52f6_4462_90cf_dd806344c29a.slice/crio-d990d9e714d0a8c479556a1c8cf8a89c54d9b24830ac948538a0f51bfeab62b1 WatchSource:0}: Error finding container d990d9e714d0a8c479556a1c8cf8a89c54d9b24830ac948538a0f51bfeab62b1: Status 404 returned error can't find the container with id d990d9e714d0a8c479556a1c8cf8a89c54d9b24830ac948538a0f51bfeab62b1
Oct 30 22:06:26.970637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:26.970611    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"]
Oct 30 22:06:27.753898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:27.753863    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" event={"ID":"a9f65885-52f6-4462-90cf-dd806344c29a","Type":"ContainerStarted","Data":"3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e"}
Oct 30 22:06:27.753898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:27.753907    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" event={"ID":"a9f65885-52f6-4462-90cf-dd806344c29a","Type":"ContainerStarted","Data":"d990d9e714d0a8c479556a1c8cf8a89c54d9b24830ac948538a0f51bfeab62b1"}
Oct 30 22:06:27.878352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:27.878300    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podStartSLOduration=1.878285779 podStartE2EDuration="1.878285779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:27.874999509 +0000 UTC m=+999.399142273" watchObservedRunningTime="2026-10-30 22:06:27.878285779 +0000 UTC m=+999.402428545"
Oct 30 22:06:28.533304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:28.533272    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:29.886815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:29.886559    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:31.314119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:31.314078    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:35.710788 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:35.710753    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:36.526954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:36.526917    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:36.550160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:36.550126    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:36.783297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:36.775630    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:37.118862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:37.118542    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:06:41.700686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:41.700614    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:44.415427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:44.415385    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:45.511065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.511033    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"]
Oct 30 22:06:45.515786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.515759    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.611697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.611663    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.611949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.611765    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.611949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.611799    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.611949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.611823    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.611949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.611868    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.611949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.611905    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2bt\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-kube-api-access-2s2bt\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.611949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.611931    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.612327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.612260    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd502b3-3610-41ce-b58a-9e028f86e765-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.612385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.612366    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.627286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.627255    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"]
Oct 30 22:06:45.713704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713671    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.713907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713730    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.713907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713750    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.713907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713767    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.713907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713805    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.713907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713841    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2bt\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-kube-api-access-2s2bt\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.713907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713869    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.714272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.713946    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd502b3-3610-41ce-b58a-9e028f86e765-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.714272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.714237    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.714272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.714242    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.714272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.714202    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.714486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.714419    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.714523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.714503    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.714926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.714865    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd502b3-3610-41ce-b58a-9e028f86e765-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.716340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.716317    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.716826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.716802    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.779985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.779914    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.780170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.780145    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2bt\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-kube-api-access-2s2bt\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:45.798203 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.798167    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:45.830515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:45.830476    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:46.288280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:46.288247    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"]
Oct 30 22:06:46.322012 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:06:46.321970    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd502b3_3610_41ce_b58a_9e028f86e765.slice/crio-4f833482007d8eeb761d12b6ad4ce4de74c5ec46b91434c5ee95e0da9d5f68d9 WatchSource:0}: Error finding container 4f833482007d8eeb761d12b6ad4ce4de74c5ec46b91434c5ee95e0da9d5f68d9: Status 404 returned error can't find the container with id 4f833482007d8eeb761d12b6ad4ce4de74c5ec46b91434c5ee95e0da9d5f68d9
Oct 30 22:06:46.324621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:46.324587    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.324700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:46.324655    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.324700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:46.324686    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:06:46.799789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:46.799748    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:46.973283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:46.973245    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" event={"ID":"0dd502b3-3610-41ce-b58a-9e028f86e765","Type":"ContainerStarted","Data":"40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607"}
Oct 30 22:06:46.973474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:46.973310    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" event={"ID":"0dd502b3-3610-41ce-b58a-9e028f86e765","Type":"ContainerStarted","Data":"4f833482007d8eeb761d12b6ad4ce4de74c5ec46b91434c5ee95e0da9d5f68d9"}
Oct 30 22:06:47.127747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:47.127674    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podStartSLOduration=2.127649517 podStartE2EDuration="2.127649517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.102285717 +0000 UTC m=+1018.626428498" watchObservedRunningTime="2026-10-30 22:06:47.127649517 +0000 UTC m=+1018.651792282"
Oct 30 22:06:47.831525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:47.831477    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:47.833760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:47.833732    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.833880 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:47.833795    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 22:06:48.831652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:48.831610    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.832082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:48.831669    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 22:06:49.832272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:49.831720    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:06:49.832272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:49.831797    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 22:06:50.557270 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:06:50.557201    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:06:50.831848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:50.831731    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:06:50.831848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:50.831807    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 22:06:51.505542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.505509    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"]
Oct 30 22:06:51.536672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.536634    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.568877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.568848    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"]
Oct 30 22:06:51.630923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.630575    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"openshift-service-ca.crt\""
Oct 30 22:06:51.630923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.630723    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"kube-root-ca.crt\""
Oct 30 22:06:51.630923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.630813    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"distinct-w3-8ca52a-kserve-self-signed-certs\""
Oct 30 22:06:51.630923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.630882    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"default-dockercfg-7cw79\""
Oct 30 22:06:51.637938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.637903    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-model-cache\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.638116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.638011    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-dshm\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.638116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.638035    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ccb0670-ee31-4846-9e1c-7a1738e35208-tls-certs\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.638302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.638173    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdxt\" (UniqueName: \"kubernetes.io/projected/6ccb0670-ee31-4846-9e1c-7a1738e35208-kube-api-access-prdxt\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.638302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.638242    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-tmp-dir\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.638445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.638366    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-home\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.670232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.670085    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:51.741361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.741289    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-dshm\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.741851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.741820    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ccb0670-ee31-4846-9e1c-7a1738e35208-tls-certs\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.741981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.741897    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdxt\" (UniqueName: \"kubernetes.io/projected/6ccb0670-ee31-4846-9e1c-7a1738e35208-kube-api-access-prdxt\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.741981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.741938    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-tmp-dir\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.742119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.742098    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-home\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.742454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.742419    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-model-cache\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.742995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.742934    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-model-cache\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.743471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.743446    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-home\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.744549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.744511    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-tmp-dir\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.751129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.746117    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ccb0670-ee31-4846-9e1c-7a1738e35208-tls-certs\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.751533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.751503    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-dshm\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.836316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.835026    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdxt\" (UniqueName: \"kubernetes.io/projected/6ccb0670-ee31-4846-9e1c-7a1738e35208-kube-api-access-prdxt\") pod \"distinct-w3-8ca52a-kserve-5779784bb4-nx5hq\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") " pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:51.836316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.835586    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:06:51.836316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.835645    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 22:06:51.852329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:51.852299    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:52.280969 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:06:52.280937    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ccb0670_ee31_4846_9e1c_7a1738e35208.slice/crio-936b39eba72de59deda2f049c68cfa1e51e002c660239179437c1fd330ac4bcc WatchSource:0}: Error finding container 936b39eba72de59deda2f049c68cfa1e51e002c660239179437c1fd330ac4bcc: Status 404 returned error can't find the container with id 936b39eba72de59deda2f049c68cfa1e51e002c660239179437c1fd330ac4bcc
Oct 30 22:06:52.281896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:52.281870    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"]
Oct 30 22:06:52.471616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:52.471565    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:52.831411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:52.831376    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:06:52.831843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:52.831431    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 22:06:53.062850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:53.062808    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" event={"ID":"6ccb0670-ee31-4846-9e1c-7a1738e35208","Type":"ContainerStarted","Data":"2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248"}
Oct 30 22:06:53.063054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:53.062863    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" event={"ID":"6ccb0670-ee31-4846-9e1c-7a1738e35208","Type":"ContainerStarted","Data":"936b39eba72de59deda2f049c68cfa1e51e002c660239179437c1fd330ac4bcc"}
Oct 30 22:06:53.192291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:53.192233    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podStartSLOduration=2.192219292 podStartE2EDuration="2.192219292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:53.192171958 +0000 UTC m=+1024.716314740" watchObservedRunningTime="2026-10-30 22:06:53.192219292 +0000 UTC m=+1024.716362047"
Oct 30 22:06:53.836141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:53.836108    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:53.836668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:53.836583    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:53.837470 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:53.837450    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:06:53.853288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:53.853263    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:06:55.183861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.183821    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-skzk8-jft2f"]
Oct 30 22:06:55.188610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.188584    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.249332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.249278    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-skzk8-jft2f"]
Oct 30 22:06:55.292509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.292473    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qmkfs\""
Oct 30 22:06:55.331054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.331007    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-archives-path\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.331291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.331100    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-service-ca-bundle\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.331364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.331335    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-trusted-ca-bundle\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.331469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.331452    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmc6\" (UniqueName: \"kubernetes.io/projected/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-kube-api-access-ggmc6\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.432921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.432883    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-trusted-ca-bundle\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.433119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.432985    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmc6\" (UniqueName: \"kubernetes.io/projected/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-kube-api-access-ggmc6\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.433119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.433082    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-archives-path\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.433314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.433137    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-service-ca-bundle\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.433749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.433716    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-archives-path\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.433995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.433945    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-trusted-ca-bundle\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.433995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.433976    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-service-ca-bundle\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.496610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.496578    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmc6\" (UniqueName: \"kubernetes.io/projected/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-kube-api-access-ggmc6\") pod \"periodic-gathering-skzk8-jft2f\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") " pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.504451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.504424    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:06:55.901190 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:06:55.901159    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95dc5cf_b483_4c3f_8a5b_7ee2691e1282.slice/crio-037c5e2dfae23b6506904d67c25fb60b52f957a253b3f90fd9e180eced9ac692 WatchSource:0}: Error finding container 037c5e2dfae23b6506904d67c25fb60b52f957a253b3f90fd9e180eced9ac692: Status 404 returned error can't find the container with id 037c5e2dfae23b6506904d67c25fb60b52f957a253b3f90fd9e180eced9ac692
Oct 30 22:06:55.901919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:55.901891    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-skzk8-jft2f"]
Oct 30 22:06:56.087113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:56.087078    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-skzk8-jft2f" event={"ID":"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282","Type":"ContainerStarted","Data":"8bb9b9e543718d34178e8337b8c30b46746b5bf69d077a1a67cb8af00d327341"}
Oct 30 22:06:56.087338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:56.087150    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-skzk8-jft2f" event={"ID":"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282","Type":"ContainerStarted","Data":"037c5e2dfae23b6506904d67c25fb60b52f957a253b3f90fd9e180eced9ac692"}
Oct 30 22:06:56.212690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:56.212523    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-skzk8-jft2f" podStartSLOduration=1.212504912 podStartE2EDuration="1.212504912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:56.206459206 +0000 UTC m=+1027.730601980" watchObservedRunningTime="2026-10-30 22:06:56.212504912 +0000 UTC m=+1027.736647678"
Oct 30 22:06:57.599322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:57.599177    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.118370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:06:59.118334    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:00.078898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:00.078849    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:07:01.859724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:01.856646    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:07:01.874044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:01.874009    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:07:02.205052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:02.204966    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:07:02.731940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:02.731894    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:07.820839 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:07.820801    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:12.914130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:12.914090    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.121587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.121551    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"]
Oct 30 22:07:17.126282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.126265    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.222533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.222498    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"]
Oct 30 22:07:17.240011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.239973    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240024    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240258    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240322    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240434    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs97h\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-kube-api-access-qs97h\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240474    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240549    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240587    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.240694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.240684    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341361    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341434    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs97h\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-kube-api-access-qs97h\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341455    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341486    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341508    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341555    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341581    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341600    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341733    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341803    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.341890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.341812    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.342052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.342037    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.342141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.342119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.342484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.342467    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.344387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.344361    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.344516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.344483    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.405344 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.405273    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.405552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.405528    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs97h\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-kube-api-access-qs97h\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.440765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.440730    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:17.627171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.627139    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"]
Oct 30 22:07:17.632980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.632958    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.644503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.644474    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.644611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.644514    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.644684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.644645    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.644684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.644672    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfm8\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-kube-api-access-ctfm8\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.644829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.644802    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.644926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.644909    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.644973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.644939    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.645033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.645022    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249019e3-0f9c-4671-9b57-d629ee30cd6b-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.645076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.645054    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.688431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.688356    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9cff60-openshift-default-dockercfg-z696x\""
Oct 30 22:07:17.693885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.693862    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"]
Oct 30 22:07:17.746085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746054    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746093    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfm8\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-kube-api-access-ctfm8\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746122    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746166    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746186    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746290    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249019e3-0f9c-4671-9b57-d629ee30cd6b-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746314    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746379    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746407    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746659    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746700    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746750    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.746877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.746853    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.747101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.747076    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249019e3-0f9c-4671-9b57-d629ee30cd6b-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.748799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.748782    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.749042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.749023    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.810199 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.810170    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.810345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.810328    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfm8\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-kube-api-access-ctfm8\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:17.919312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.919272    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:17.932103 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:07:17.932067    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode337b5e1_86c2_4f6e_bb6c_afcfa9e47f73.slice/crio-d658ce79851566723eb1da5641f4c06509d55abf1944fe4073fced4471a577e6 WatchSource:0}: Error finding container d658ce79851566723eb1da5641f4c06509d55abf1944fe4073fced4471a577e6: Status 404 returned error can't find the container with id d658ce79851566723eb1da5641f4c06509d55abf1944fe4073fced4471a577e6
Oct 30 22:07:17.935049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.935019    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"]
Oct 30 22:07:17.936038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.935993    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:07:17.936132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.936108    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:07:17.936318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.936296    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:07:17.952982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:17.948728    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:18.342678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.342633    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" event={"ID":"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73","Type":"ContainerStarted","Data":"5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7"}
Oct 30 22:07:18.343153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.342712    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" event={"ID":"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73","Type":"ContainerStarted","Data":"d658ce79851566723eb1da5641f4c06509d55abf1944fe4073fced4471a577e6"}
Oct 30 22:07:18.395015 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:07:18.394975    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249019e3_0f9c_4671_9b57_d629ee30cd6b.slice/crio-c87ef66d3a5076f597aba7b42b6a67dc817bc8e90d465c76a99752ae4bdce144 WatchSource:0}: Error finding container c87ef66d3a5076f597aba7b42b6a67dc817bc8e90d465c76a99752ae4bdce144: Status 404 returned error can't find the container with id c87ef66d3a5076f597aba7b42b6a67dc817bc8e90d465c76a99752ae4bdce144
Oct 30 22:07:18.398760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.398732    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"]
Oct 30 22:07:18.399945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.399906    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:07:18.400048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.399999    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:07:18.400104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.400051    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:07:18.441359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.441299    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:18.471568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:18.471504    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podStartSLOduration=1.471482656 podStartE2EDuration="1.471482656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:18.467987904 +0000 UTC m=+1049.992130696" watchObservedRunningTime="2026-10-30 22:07:18.471482656 +0000 UTC m=+1049.995625424"
Oct 30 22:07:19.368950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.368910    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" event={"ID":"249019e3-0f9c-4671-9b57-d629ee30cd6b","Type":"ContainerStarted","Data":"79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0"}
Oct 30 22:07:19.369488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.368972    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" event={"ID":"249019e3-0f9c-4671-9b57-d629ee30cd6b","Type":"ContainerStarted","Data":"c87ef66d3a5076f597aba7b42b6a67dc817bc8e90d465c76a99752ae4bdce144"}
Oct 30 22:07:19.458247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.457511    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:07:19.458247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.457578    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 22:07:19.571127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.571066    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podStartSLOduration=2.5710490200000002 podStartE2EDuration="2.57104902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:19.548852306 +0000 UTC m=+1051.072995094" watchObservedRunningTime="2026-10-30 22:07:19.57104902 +0000 UTC m=+1051.095191784"
Oct 30 22:07:19.962328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.962283    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:19.963448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.963418    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:07:19.963565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:19.963475    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 22:07:20.443231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:20.442973    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:07:20.443231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:20.443034    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 22:07:20.584096 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:07:20.583787    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:07:20.949673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:20.949632    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:07:20.949989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:20.949955    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 22:07:21.447239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:21.443905    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:07:21.447239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:21.443965    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 22:07:21.954421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:21.954386    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:07:21.954732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:21.954701    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 22:07:22.450245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:22.442443    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:07:22.450245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:22.442503    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 22:07:22.951647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:22.951601    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:07:22.951884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:22.951669    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 22:07:23.015233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:23.015060    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:23.443421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:23.443388    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:07:23.443696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:23.443667    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 22:07:23.957659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:23.957620    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:07:23.958288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:23.958256    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 22:07:24.177023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:24.176988    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:24.400326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:24.400293    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:24.448244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:24.447492    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:07:24.448244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:24.447552    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 22:07:24.949634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:24.949592    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:07:24.949935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:24.949900    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 22:07:25.447503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:25.447474    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:25.448266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:25.448240    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:25.449062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:25.449036    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:07:25.956782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:25.956752    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:25.957589 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:25.957566    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:25.958421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:25.958405    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:07:28.098817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:28.098784    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:31.014729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:31.014689    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:31.242287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:31.242201    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:33.132818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:33.132789    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:33.797353 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:33.797302    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:38.237984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:38.237945    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:41.990908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:41.990875    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:43.219444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:43.219405    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:43.559071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:43.558987    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:07:44.740366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:44.740332    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:48.302165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:48.302127    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:50.542755 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:07:50.542717    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:07:53.401404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:53.401341    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:58.507916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:58.507883    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:59.325008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:07:59.324970    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:02.727762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.727724    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"]
Oct 30 22:08:02.732143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.732123    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.778109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.778080    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"]
Oct 30 22:08:02.782560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.782543    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.796650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.796623    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.796773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.796675    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0090e71b-05c6-47b4-a868-6f997213b01d-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.796773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.796743    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.796858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.796829    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqxq\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-kube-api-access-fhqxq\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.796900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.796890    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.796953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.796908    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.796953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.796941    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.797127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.797110    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.797259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.797160    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0090e71b-05c6-47b4-a868-6f997213b01d-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.867229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.867178    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"]
Oct 30 22:08:02.898752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.898722    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0090e71b-05c6-47b4-a868-6f997213b01d-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.898943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.898763    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.898943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.898859    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.898943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.898923    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.899074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899004    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899033    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.899074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899069    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ssh\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-kube-api-access-98ssh\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.899254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899157    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.899254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899195    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0090e71b-05c6-47b4-a868-6f997213b01d-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899327    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899342    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899406    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.899457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899424    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0090e71b-05c6-47b4-a868-6f997213b01d-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899547    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqxq\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-kube-api-access-fhqxq\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899630    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.899718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899681    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899706    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899722    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.899829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899772    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:02.899935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.899906    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.900427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.900403    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.900527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.900421    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.900920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.900893    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.901908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.901882    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.902045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.902024    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0090e71b-05c6-47b4-a868-6f997213b01d-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.941666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.941636    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"]
Oct 30 22:08:02.969372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.969340    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:02.972565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:02.972425    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqxq\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-kube-api-access-fhqxq\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:03.001591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.001507    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.001591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.001550    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.001814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.001611    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.001814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.001656    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.001814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.001762    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.001968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.001862    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002202    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002269    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002356    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002362    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002403    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ssh\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-kube-api-access-98ssh\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002443    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002522    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.002655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.002618    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.004533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.004512    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.004745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.004728    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.047299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.047269    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:03.068283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.068254    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.068504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.068484    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ssh\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-kube-api-access-98ssh\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.099022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.098990    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:03.499573 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:08:03.499540    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0090e71b_05c6_47b4_a868_6f997213b01d.slice/crio-00c4bc34fce2c2b4caa00d0d64c0f9cc9ec41a79ef2a5ecc7c9bdc4c575ecd27 WatchSource:0}: Error finding container 00c4bc34fce2c2b4caa00d0d64c0f9cc9ec41a79ef2a5ecc7c9bdc4c575ecd27: Status 404 returned error can't find the container with id 00c4bc34fce2c2b4caa00d0d64c0f9cc9ec41a79ef2a5ecc7c9bdc4c575ecd27
Oct 30 22:08:03.500093 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.500069    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"]
Oct 30 22:08:03.502447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.502388    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.502536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.502475    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.502536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.502507    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.544559 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.544525    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"]
Oct 30 22:08:03.546064 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:08:03.546024    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3fd37c_ce13_4aa4_8af2_b3d89621fc10.slice/crio-0bd39c799a97c35a867651f521123771a08433f39890eebf68b232265b0cb3de WatchSource:0}: Error finding container 0bd39c799a97c35a867651f521123771a08433f39890eebf68b232265b0cb3de: Status 404 returned error can't find the container with id 0bd39c799a97c35a867651f521123771a08433f39890eebf68b232265b0cb3de
Oct 30 22:08:03.549091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.549048    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.549197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.549137    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.549197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.549176    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:03.647912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.640403    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:03.785941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.785844    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" event={"ID":"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10","Type":"ContainerStarted","Data":"917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39"}
Oct 30 22:08:03.785941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.785920    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" event={"ID":"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10","Type":"ContainerStarted","Data":"0bd39c799a97c35a867651f521123771a08433f39890eebf68b232265b0cb3de"}
Oct 30 22:08:03.788309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.788276    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" event={"ID":"0090e71b-05c6-47b4-a868-6f997213b01d","Type":"ContainerStarted","Data":"50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753"}
Oct 30 22:08:03.788494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.788328    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" event={"ID":"0090e71b-05c6-47b4-a868-6f997213b01d","Type":"ContainerStarted","Data":"00c4bc34fce2c2b4caa00d0d64c0f9cc9ec41a79ef2a5ecc7c9bdc4c575ecd27"}
Oct 30 22:08:03.916898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:03.916821    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podStartSLOduration=1.916798596 podStartE2EDuration="1.916798596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:03.912049538 +0000 UTC m=+1095.436192309" watchObservedRunningTime="2026-10-30 22:08:03.916798596 +0000 UTC m=+1095.440941366"
Oct 30 22:08:04.047838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:04.047724    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podStartSLOduration=2.047708897 podStartE2EDuration="2.047708897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:04.0408351 +0000 UTC m=+1095.564977864" watchObservedRunningTime="2026-10-30 22:08:04.047708897 +0000 UTC m=+1095.571851661"
Oct 30 22:08:04.048283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:04.048261    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:04.050052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:04.050024    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:08:04.050166 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:04.050081    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:08:04.100611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:04.100547    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:04.105164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:04.104340    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:08:04.105164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:04.104405    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:08:05.050236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:05.049659    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:08:05.050236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:05.049724    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:08:05.100736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:05.100700    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:08:05.100934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:05.100757    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:08:06.049233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:06.048660    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:08:06.049233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:06.048726    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:08:06.099989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:06.099958    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:08:06.100492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:06.100466    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:08:07.052687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:07.052645    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:08:07.052995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:07.052963    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:08:07.100185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:07.100153    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:08:07.100592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:07.100234    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:08:08.049619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:08.049579    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:08:08.049924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:08.049898    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:08:08.099363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:08.099325    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:08:08.099595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:08.099574    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:08:08.741478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:08.741439    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:09.049736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:09.049639    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:08:09.050018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:09.049985    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:08:09.100006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:09.099975    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:08:09.100180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:09.100027    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:08:10.048124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:10.048077    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:08:10.048695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:10.048147    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:08:10.099879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:10.099847    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:08:10.100066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:10.099903    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:08:11.089653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:11.089618    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:11.090567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:11.090544    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:11.091117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:11.091094    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:08:11.103640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:11.103614    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:11.103901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:11.103883    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:11.104673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:11.104658    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:08:13.802382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:13.802294    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:18.923607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:18.923559    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:20.542990 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:08:20.542946    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:08:23.929731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:23.929696    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:29.010765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:29.010675    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:31.406285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.406248    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"]
Oct 30 22:08:31.412161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.412137    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413016    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413062    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nh5k\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-kube-api-access-2nh5k\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413170    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413197    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413350    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413407    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413469    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413556    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.413644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.413626    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.514923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.514882    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.514962    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515029    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515094    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515158    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515268    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515298    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nh5k\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-kube-api-access-2nh5k\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515367    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515388    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515395    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515758    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.515851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.515800    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.519246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.516302    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.519246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.516414    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.521733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.521687    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.521833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.521754    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.532381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.532355    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"]
Oct 30 22:08:31.579602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.579570    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.579773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.579756    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nh5k\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-kube-api-access-2nh5k\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:31.729260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:31.729155    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:32.121992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:32.121961    5031 generic.go:356] "Generic (PLEG): container finished" podID="f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" containerID="8bb9b9e543718d34178e8337b8c30b46746b5bf69d077a1a67cb8af00d327341" exitCode=0
Oct 30 22:08:32.122163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:32.122038    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-skzk8-jft2f" event={"ID":"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282","Type":"ContainerDied","Data":"8bb9b9e543718d34178e8337b8c30b46746b5bf69d077a1a67cb8af00d327341"}
Oct 30 22:08:32.174744 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:08:32.174717    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f61a9c_f88c_4bad_baa0_b630b6e6184e.slice/crio-f599c35cf6585c6379fcd1311f1a47cb333ef0fec0622809f791b47160c028da WatchSource:0}: Error finding container f599c35cf6585c6379fcd1311f1a47cb333ef0fec0622809f791b47160c028da: Status 404 returned error can't find the container with id f599c35cf6585c6379fcd1311f1a47cb333ef0fec0622809f791b47160c028da
Oct 30 22:08:32.175652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:32.175629    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"]
Oct 30 22:08:32.177348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:32.177320    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:32.177417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:32.177383    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:32.177417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:32.177412    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:08:33.136585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.136546    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" event={"ID":"82f61a9c-f88c-4bad-baa0-b630b6e6184e","Type":"ContainerStarted","Data":"c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6"}
Oct 30 22:08:33.137018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.136616    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" event={"ID":"82f61a9c-f88c-4bad-baa0-b630b6e6184e","Type":"ContainerStarted","Data":"f599c35cf6585c6379fcd1311f1a47cb333ef0fec0622809f791b47160c028da"}
Oct 30 22:08:33.350022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.349967    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podStartSLOduration=2.349952077 podStartE2EDuration="2.349952077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.345653296 +0000 UTC m=+1124.869796077" watchObservedRunningTime="2026-10-30 22:08:33.349952077 +0000 UTC m=+1124.874094923"
Oct 30 22:08:33.394169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.394108    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:08:33.555871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.555832    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-archives-path\") pod \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") "
Oct 30 22:08:33.556056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.555897    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-trusted-ca-bundle\") pod \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") "
Oct 30 22:08:33.556056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.555965    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-kube-api-access-ggmc6\" (UniqueName: \"kubernetes.io/projected/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-kube-api-access-ggmc6\") pod \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") "
Oct 30 22:08:33.556056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.556009    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-service-ca-bundle\") pod \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\" (UID: \"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282\") "
Oct 30 22:08:33.556820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.556790    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-trusted-ca-bundle" pod "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" (UID: "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:33.557013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.556992    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-archives-path" pod "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" (UID: "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:33.557259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.557234    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-service-ca-bundle" pod "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" (UID: "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:33.557550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.557530    5031 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-archives-path\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.557637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.557553    5031 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-trusted-ca-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.559131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.559111    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-kube-api-access-ggmc6" pod "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" (UID: "f95dc5cf-b483-4c3f-8a5b-7ee2691e1282"). InnerVolumeSpecName "kube-api-access-ggmc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:33.659295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.659018    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggmc6\" (UniqueName: \"kubernetes.io/projected/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-kube-api-access-ggmc6\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.659295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.659044    5031 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f95dc5cf-b483-4c3f-8a5b-7ee2691e1282-service-ca-bundle\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.737275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.734199    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:33.737275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.734366    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:08:33.737275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:33.734431    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:08:34.149789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:34.149757    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-skzk8-jft2f"
Oct 30 22:08:34.149789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:34.149787    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-skzk8-jft2f" event={"ID":"f95dc5cf-b483-4c3f-8a5b-7ee2691e1282","Type":"ContainerDied","Data":"037c5e2dfae23b6506904d67c25fb60b52f957a253b3f90fd9e180eced9ac692"}
Oct 30 22:08:34.150328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:34.149833    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="037c5e2dfae23b6506904d67c25fb60b52f957a253b3f90fd9e180eced9ac692"
Oct 30 22:08:34.259533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:34.259234    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:34.733235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:34.729664    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:08:34.733235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:34.729744    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:08:35.247283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:35.247237    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:35.481077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:35.481045    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:35.730235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:35.729616    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:08:35.730235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:35.729694    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:08:36.730234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:36.729609    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:08:36.730234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:36.729685    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:08:36.915782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:36.915744    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:37.732650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:37.732613    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:08:37.733277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:37.733242    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:08:38.730321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:38.730283    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:08:38.730533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:38.730340    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:08:38.850657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:38.850613    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:08:39.517959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:39.517919    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:39.958087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:39.958052    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:39.958546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:39.958466    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:39.959198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:39.959178    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:08:41.075725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:41.075628    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"]
Oct 30 22:08:41.128541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:41.128508    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e454ce-cxjv4"]
Oct 30 22:08:42.649657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:42.649611    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f" path="/var/lib/kubelet/pods/e81eadeb-a6f1-4fd1-9db9-e72a9bbf165f/volumes"
Oct 30 22:08:44.495802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:44.495769    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:45.096640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:45.096606    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:45.389711 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:45.389627    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:45.835611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:45.835496    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:49.537352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.537308    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:49.876436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.876404    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"]
Oct 30 22:08:49.877115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.877101    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" containerName="insights-gathering"
Oct 30 22:08:49.877166 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.877117    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" containerName="insights-gathering"
Oct 30 22:08:49.877257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.877247    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95dc5cf-b483-4c3f-8a5b-7ee2691e1282" containerName="insights-gathering"
Oct 30 22:08:49.881375 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.881358    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:49.940549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.940521    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"]
Oct 30 22:08:49.983958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.983928    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:49.984157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.983992    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:49.984157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.984045    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:49.985515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:49.985500    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:08:50.009281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.009253    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92738c9a-624f-4658-8071-40de3b792487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.009436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.009307    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92738c9a-624f-4658-8071-40de3b792487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.009502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.009438    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzl5q\" (UniqueName: \"kubernetes.io/projected/92738c9a-624f-4658-8071-40de3b792487-kube-api-access-dzl5q\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.110648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.110610    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92738c9a-624f-4658-8071-40de3b792487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.110833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.110673    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92738c9a-624f-4658-8071-40de3b792487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.110833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.110763    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzl5q\" (UniqueName: \"kubernetes.io/projected/92738c9a-624f-4658-8071-40de3b792487-kube-api-access-dzl5q\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.111334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.111312    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92738c9a-624f-4658-8071-40de3b792487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.113355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.113329    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92738c9a-624f-4658-8071-40de3b792487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.174802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.174734    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzl5q\" (UniqueName: \"kubernetes.io/projected/92738c9a-624f-4658-8071-40de3b792487-kube-api-access-dzl5q\") pod \"maas-api-revoke-sub-14363982-rx8gj\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.196846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.196819    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:50.542664 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:08:50.542574    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:08:50.583564 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:08:50.583533    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92738c9a_624f_4658_8071_40de3b792487.slice/crio-72e33177818f28676422ce23303d1f127945aff54f6c1132c7743f5da5f1688f WatchSource:0}: Error finding container 72e33177818f28676422ce23303d1f127945aff54f6c1132c7743f5da5f1688f: Status 404 returned error can't find the container with id 72e33177818f28676422ce23303d1f127945aff54f6c1132c7743f5da5f1688f
Oct 30 22:08:50.586447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:50.586418    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"]
Oct 30 22:08:51.305657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:51.305615    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj" event={"ID":"92738c9a-624f-4658-8071-40de3b792487","Type":"ContainerStarted","Data":"4f43625f81b5aca505ac67c7aa01b6f1bbb2e5ff2e43ae6eb2f54b5e2d9aeec1"}
Oct 30 22:08:51.305857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:51.305711    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj" event={"ID":"92738c9a-624f-4658-8071-40de3b792487","Type":"ContainerStarted","Data":"72e33177818f28676422ce23303d1f127945aff54f6c1132c7743f5da5f1688f"}
Oct 30 22:08:51.435697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:51.435634    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj" podStartSLOduration=2.4356167060000002 podStartE2EDuration="2.435616706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:51.427663517 +0000 UTC m=+1142.951806283" watchObservedRunningTime="2026-10-30 22:08:51.435616706 +0000 UTC m=+1142.959759471"
Oct 30 22:08:52.316908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:52.316867    5031 generic.go:356] "Generic (PLEG): container finished" podID="92738c9a-624f-4658-8071-40de3b792487" containerID="4f43625f81b5aca505ac67c7aa01b6f1bbb2e5ff2e43ae6eb2f54b5e2d9aeec1" exitCode=0
Oct 30 22:08:52.317425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:52.316985    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj" event={"ID":"92738c9a-624f-4658-8071-40de3b792487","Type":"ContainerDied","Data":"4f43625f81b5aca505ac67c7aa01b6f1bbb2e5ff2e43ae6eb2f54b5e2d9aeec1"}
Oct 30 22:08:53.509284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.509249    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:53.560888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.560859    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92738c9a-624f-4658-8071-40de3b792487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92738c9a-624f-4658-8071-40de3b792487-maas-api-xks-ca\") pod \"92738c9a-624f-4658-8071-40de3b792487\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") "
Oct 30 22:08:53.561064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.561025    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92738c9a-624f-4658-8071-40de3b792487-kube-api-access-dzl5q\" (UniqueName: \"kubernetes.io/projected/92738c9a-624f-4658-8071-40de3b792487-kube-api-access-dzl5q\") pod \"92738c9a-624f-4658-8071-40de3b792487\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") "
Oct 30 22:08:53.561064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.561057    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92738c9a-624f-4658-8071-40de3b792487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92738c9a-624f-4658-8071-40de3b792487-maas-api-openshift-ca\") pod \"92738c9a-624f-4658-8071-40de3b792487\" (UID: \"92738c9a-624f-4658-8071-40de3b792487\") "
Oct 30 22:08:53.561527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.561497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92738c9a-624f-4658-8071-40de3b792487-maas-api-openshift-ca" pod "92738c9a-624f-4658-8071-40de3b792487" (UID: "92738c9a-624f-4658-8071-40de3b792487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:53.563238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.563194    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92738c9a-624f-4658-8071-40de3b792487-maas-api-xks-ca" pod "92738c9a-624f-4658-8071-40de3b792487" (UID: "92738c9a-624f-4658-8071-40de3b792487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:53.563456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.563435    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92738c9a-624f-4658-8071-40de3b792487-kube-api-access-dzl5q" pod "92738c9a-624f-4658-8071-40de3b792487" (UID: "92738c9a-624f-4658-8071-40de3b792487"). InnerVolumeSpecName "kube-api-access-dzl5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:53.662716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.662683    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92738c9a-624f-4658-8071-40de3b792487-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:53.662716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.662713    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzl5q\" (UniqueName: \"kubernetes.io/projected/92738c9a-624f-4658-8071-40de3b792487-kube-api-access-dzl5q\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:53.662918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:53.662727    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92738c9a-624f-4658-8071-40de3b792487-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.337622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:54.337591    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"
Oct 30 22:08:54.337622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:54.337608    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj" event={"ID":"92738c9a-624f-4658-8071-40de3b792487","Type":"ContainerDied","Data":"72e33177818f28676422ce23303d1f127945aff54f6c1132c7743f5da5f1688f"}
Oct 30 22:08:54.337870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:54.337659    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72e33177818f28676422ce23303d1f127945aff54f6c1132c7743f5da5f1688f"
Oct 30 22:08:54.590014 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:54.589923    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:56.337266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:56.337199    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:57.069834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:57.069794    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"]
Oct 30 22:08:57.120323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:57.120284    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546bcf35-m48x7"]
Oct 30 22:08:58.653252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:58.653192    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2659a03-b3c3-4ef5-8735-ff7b277d811f" path="/var/lib/kubelet/pods/c2659a03-b3c3-4ef5-8735-ff7b277d811f/volumes"
Oct 30 22:08:59.858777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:08:59.858739    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:04.928594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:04.928557    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:05.059898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:05.059862    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"]
Oct 30 22:09:05.110433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:05.110399    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cba0bf-tl49c"]
Oct 30 22:09:06.177258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.177201    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"]
Oct 30 22:09:06.179364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.179303    5031 cpu_manager.go:414] "containerMap: removing container" podUID="92738c9a-624f-4658-8071-40de3b792487" containerName="revoke-keys"
Oct 30 22:09:06.179555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.179539    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92738c9a-624f-4658-8071-40de3b792487" containerName="revoke-keys"
Oct 30 22:09:06.180031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.180013    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="92738c9a-624f-4658-8071-40de3b792487" containerName="revoke-keys"
Oct 30 22:09:06.185808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.185788    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.238933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.238905    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"]
Oct 30 22:09:06.283520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.283493    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce513\"/\"distinct-2-w4-bce513-kserve-self-signed-certs\""
Oct 30 22:09:06.323659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.323597    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-dshm\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.324154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.324131    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-tmp-dir\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.324416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.324359    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-home\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.324695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.324678    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-model-cache\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.324866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.324852    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1241795d-784e-4ab5-8b92-66d23a747f3e-tls-certs\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.325064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.325047    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5t9n\" (UniqueName: \"kubernetes.io/projected/1241795d-784e-4ab5-8b92-66d23a747f3e-kube-api-access-w5t9n\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.336769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.336732    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:06.426518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.426490    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-home\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.426699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.426591    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-model-cache\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.426699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.426615    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1241795d-784e-4ab5-8b92-66d23a747f3e-tls-certs\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.426699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.426645    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5t9n\" (UniqueName: \"kubernetes.io/projected/1241795d-784e-4ab5-8b92-66d23a747f3e-kube-api-access-w5t9n\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.426813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.426705    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-dshm\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.426813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.426789    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-tmp-dir\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.426982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.426960    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-home\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.427117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.427097    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-tmp-dir\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.427573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.427511    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-model-cache\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.429788 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.429765    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-dshm\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.429894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.429804    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1241795d-784e-4ab5-8b92-66d23a747f3e-tls-certs\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.490774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.490742    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5t9n\" (UniqueName: \"kubernetes.io/projected/1241795d-784e-4ab5-8b92-66d23a747f3e-kube-api-access-w5t9n\") pod \"distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") " pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.502372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.502349    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:06.670011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.669978    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7426b215-5e4f-4391-bcbb-a5fbd38422ce" path="/var/lib/kubelet/pods/7426b215-5e4f-4391-bcbb-a5fbd38422ce/volumes"
Oct 30 22:09:06.977095 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:06.977059    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1241795d_784e_4ab5_8b92_66d23a747f3e.slice/crio-08c77629bdc74242bd661293080e183dd482f3a2c8d8d0b6284e127110520027 WatchSource:0}: Error finding container 08c77629bdc74242bd661293080e183dd482f3a2c8d8d0b6284e127110520027: Status 404 returned error can't find the container with id 08c77629bdc74242bd661293080e183dd482f3a2c8d8d0b6284e127110520027
Oct 30 22:09:06.977759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:06.977734    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"]
Oct 30 22:09:07.647904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:07.647868    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" event={"ID":"1241795d-784e-4ab5-8b92-66d23a747f3e","Type":"ContainerStarted","Data":"55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174"}
Oct 30 22:09:07.648442 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:07.647915    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" event={"ID":"1241795d-784e-4ab5-8b92-66d23a747f3e","Type":"ContainerStarted","Data":"08c77629bdc74242bd661293080e183dd482f3a2c8d8d0b6284e127110520027"}
Oct 30 22:09:07.777836 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:07.777787    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podStartSLOduration=1.777773035 podStartE2EDuration="1.777773035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:07.773469962 +0000 UTC m=+1159.297612737" watchObservedRunningTime="2026-10-30 22:09:07.777773035 +0000 UTC m=+1159.301915798"
Oct 30 22:09:08.503471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:08.503438    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:10.007949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:10.007913    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.300345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.300310    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"]
Oct 30 22:09:11.305051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.305028    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.406059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.406027    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-home\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.406271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.406099    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-model-cache\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.406314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.406277    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-dshm\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.406423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.406406    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-tmp-dir\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.406496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.406482    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdjw\" (UniqueName: \"kubernetes.io/projected/9f1c19fa-524b-4624-b955-1e22ab1c2184-kube-api-access-czdjw\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.406533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.406509    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1c19fa-524b-4624-b955-1e22ab1c2184-tls-certs\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.412254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.412232    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"]
Oct 30 22:09:11.413393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.413358    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"openshift-service-ca.crt\""
Oct 30 22:09:11.413656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.413639    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"kube-root-ca.crt\""
Oct 30 22:09:11.413730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.413667    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"default-dockercfg-gwrpm\""
Oct 30 22:09:11.413791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.413764    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"distinct-2-w1-9cff60-kserve-self-signed-certs\""
Oct 30 22:09:11.475068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.470295    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.514504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.511965    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdjw\" (UniqueName: \"kubernetes.io/projected/9f1c19fa-524b-4624-b955-1e22ab1c2184-kube-api-access-czdjw\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.514814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.514778    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1c19fa-524b-4624-b955-1e22ab1c2184-tls-certs\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.514968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.514908    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-home\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.515033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.514998    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-model-cache\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.515141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.515126    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-dshm\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.515328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.515305    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-tmp-dir\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.515743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.515719    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-home\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.515863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.515802    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-tmp-dir\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.516194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.516170    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-model-cache\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.517933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.517909    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-dshm\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.519530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.519489    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1c19fa-524b-4624-b955-1e22ab1c2184-tls-certs\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.577449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.577375    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdjw\" (UniqueName: \"kubernetes.io/projected/9f1c19fa-524b-4624-b955-1e22ab1c2184-kube-api-access-czdjw\") pod \"distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:11.618495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:11.618466    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:12.061746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:12.061711    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"]
Oct 30 22:09:12.070982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:12.070957    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"]
Oct 30 22:09:12.075680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:12.075653    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb37fc14-mp5cl"]
Oct 30 22:09:12.104265 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:12.104236    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f1c19fa_524b_4624_b955_1e22ab1c2184.slice/crio-6e7b126a650c145b7c4aa59bf0fbbc5d58268fcdfc9f08d9db4e73d6ae6e6784 WatchSource:0}: Error finding container 6e7b126a650c145b7c4aa59bf0fbbc5d58268fcdfc9f08d9db4e73d6ae6e6784: Status 404 returned error can't find the container with id 6e7b126a650c145b7c4aa59bf0fbbc5d58268fcdfc9f08d9db4e73d6ae6e6784
Oct 30 22:09:12.666323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:12.666285    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34fba32-6865-4772-86b8-9a706361fd27" path="/var/lib/kubelet/pods/f34fba32-6865-4772-86b8-9a706361fd27/volumes"
Oct 30 22:09:12.791451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:12.791418    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" event={"ID":"9f1c19fa-524b-4624-b955-1e22ab1c2184","Type":"ContainerStarted","Data":"56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca"}
Oct 30 22:09:12.791635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:12.791462    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" event={"ID":"9f1c19fa-524b-4624-b955-1e22ab1c2184","Type":"ContainerStarted","Data":"6e7b126a650c145b7c4aa59bf0fbbc5d58268fcdfc9f08d9db4e73d6ae6e6784"}
Oct 30 22:09:12.916233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:12.916148    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podStartSLOduration=1.916134845 podStartE2EDuration="1.916134845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:12.913720814 +0000 UTC m=+1164.437863588" watchObservedRunningTime="2026-10-30 22:09:12.916134845 +0000 UTC m=+1164.440277608"
Oct 30 22:09:13.619468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:13.619435    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:15.118752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:15.118717    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.060247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:16.060201    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"]
Oct 30 22:09:16.110517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:16.110481    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc06c38-tckjn"]
Oct 30 22:09:16.502673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:16.502626    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:16.515698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:16.515667    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:16.659709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:16.659675    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee75358-a467-4461-98ee-dfb8a1911764" path="/var/lib/kubelet/pods/bee75358-a467-4461-98ee-dfb8a1911764/volumes"
Oct 30 22:09:16.838268 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:16.838164    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:17.368426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.368389    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"]
Oct 30 22:09:17.373859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.373835    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.473566 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.473532    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"]
Oct 30 22:09:17.515520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.515490    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.515534    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69863d8f-0231-4dd2-ae46-1d13bc939df3-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.515616    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.515730    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.515777    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.515940    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.515969    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.516102    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.516355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.516150    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47zd\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-kube-api-access-d47zd\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.617760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.617721    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.617760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.617767    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.617839    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.617858    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.617908    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.617938    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47zd\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-kube-api-access-d47zd\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618070    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618110    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69863d8f-0231-4dd2-ae46-1d13bc939df3-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618167    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618254    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618312    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618660    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618693    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.618929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.618913    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69863d8f-0231-4dd2-ae46-1d13bc939df3-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.621048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.621020    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.621198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.621179    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.682502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.682469    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.682668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.682654    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47zd\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-kube-api-access-d47zd\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:17.690316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:17.690295    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:18.144401 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:18.144373    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69863d8f_0231_4dd2_ae46_1d13bc939df3.slice/crio-e50906739273e3ce527ee92536fe59eb5332dd67d0ce584d4c834d37b1f48805 WatchSource:0}: Error finding container e50906739273e3ce527ee92536fe59eb5332dd67d0ce584d4c834d37b1f48805: Status 404 returned error can't find the container with id e50906739273e3ce527ee92536fe59eb5332dd67d0ce584d4c834d37b1f48805
Oct 30 22:09:18.145323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.145261    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"]
Oct 30 22:09:18.147357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.147328    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:18.147436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.147389    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:18.147436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.147424    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:18.331559 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.331520    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"]
Oct 30 22:09:18.336805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.336781    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.408841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.408754    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"]
Oct 30 22:09:18.436635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.436582    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.437578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.437552    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.437749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.437734    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.437866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.437850    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.438047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.438020    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.438180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.438112    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.438180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.438153    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6c5x\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-kube-api-access-j6c5x\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.438357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.438199    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.438357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.438277    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.539789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.539754    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.539839    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.539867    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6c5x\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-kube-api-access-j6c5x\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.539899    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.539933    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540082    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540120    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540174    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540253    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540414    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540440    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.540768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540718    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.541456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.540824    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.541456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.541342    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.543099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.543079    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.543198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.543138    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.604618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.604582    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6c5x\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-kube-api-access-j6c5x\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.604791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.604727    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.652366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.652338    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:18.858793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.858751    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" event={"ID":"69863d8f-0231-4dd2-ae46-1d13bc939df3","Type":"ContainerStarted","Data":"2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1"}
Oct 30 22:09:18.858980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.858822    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" event={"ID":"69863d8f-0231-4dd2-ae46-1d13bc939df3","Type":"ContainerStarted","Data":"e50906739273e3ce527ee92536fe59eb5332dd67d0ce584d4c834d37b1f48805"}
Oct 30 22:09:18.986913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:18.986808    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podStartSLOduration=1.986795683 podStartE2EDuration="1.986795683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:18.983012926 +0000 UTC m=+1170.507155693" watchObservedRunningTime="2026-10-30 22:09:18.986795683 +0000 UTC m=+1170.510938446"
Oct 30 22:09:19.104124 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:19.104093    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode226a06c_caa4_401c_a4c2_f5d0abb2fcb6.slice/crio-d09145d5dc5d1eca76208ab6b28d40d9dac15f3d73748a84fb8d6e4e08a7cfed WatchSource:0}: Error finding container d09145d5dc5d1eca76208ab6b28d40d9dac15f3d73748a84fb8d6e4e08a7cfed: Status 404 returned error can't find the container with id d09145d5dc5d1eca76208ab6b28d40d9dac15f3d73748a84fb8d6e4e08a7cfed
Oct 30 22:09:19.104977 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.104953    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"]
Oct 30 22:09:19.121259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.107159    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:19.121259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.107301    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:19.121259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.107345    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:19.691319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.691269    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:19.693342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.693296    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:09:19.693501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.693357    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:09:19.854257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.853711    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:19.876897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.876857    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" event={"ID":"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6","Type":"ContainerStarted","Data":"e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70"}
Oct 30 22:09:19.877096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:19.876914    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" event={"ID":"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6","Type":"ContainerStarted","Data":"d09145d5dc5d1eca76208ab6b28d40d9dac15f3d73748a84fb8d6e4e08a7cfed"}
Oct 30 22:09:20.020296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:20.011819    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podStartSLOduration=2.011790009 podStartE2EDuration="2.011790009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:20.003393842 +0000 UTC m=+1171.527536618" watchObservedRunningTime="2026-10-30 22:09:20.011790009 +0000 UTC m=+1171.535932774"
Oct 30 22:09:20.253267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:20.253169    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:20.602953 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:09:20.602919    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:09:20.655142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:20.652973    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:20.659231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:20.658353    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:09:20.659231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:20.658410    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:09:20.691198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:20.691166    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:09:20.691401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:20.691235    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:09:21.635237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:21.631371    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:21.665235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:21.653573    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:09:21.665235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:21.653631    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:09:21.670336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:21.667857    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:21.690681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:21.690633    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:09:21.690835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:21.690695    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:09:22.001967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:22.001890    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:22.670239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:22.669371    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:09:22.670239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:22.669428    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:09:22.691045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:22.691013    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:09:22.691238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:22.691071    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:09:23.655237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:23.653804    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:09:23.655237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:23.653924    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:09:23.691309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:23.691274    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:09:23.691501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:23.691325    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:09:24.653229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:24.653166    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:09:24.654269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:24.654233    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:09:24.690917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:24.690881    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:09:24.691366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:24.690953    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:09:25.653284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:25.653243    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:09:25.653465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:25.653323    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:09:25.691592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:25.691558    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:09:25.692033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:25.691613    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:09:25.764733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:25.764694    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:26.655371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:26.655326    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:09:26.655558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:26.655390    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:09:26.696491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:26.696456    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:26.697499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:26.697472    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:26.698366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:26.698342    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:09:27.670999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:27.670969    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:27.671501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:27.671479    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:27.672202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:27.672180    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:09:28.190489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:28.190450    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:09:30.753955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:30.753901    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:30.833394 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:30.833337    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:30.918830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:30.918757    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:30.957558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:30.957512    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.014405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.014296    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.212551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.212503    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:09:31.294481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.294338    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.517255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.487199    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:09:31.528309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.518807    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:31.762375 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.762326    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.763820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.763783    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:09:31.844233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.841369    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.844233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.842172    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:31.924232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.922309    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:31.924232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:31.923108    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:09:32.095395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.094573    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:32.114596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.114552    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:09:32.133255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.116624    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:32.322496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.322445    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:09:32.406030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.377405    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:09:32.445684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.445645    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:09:32.468080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.455114    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:09:32.526381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.526176    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:09:32.529292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.527752    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:09:32.783238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.776189    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:32.786649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.786599    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:32.952397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.939290    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:32.977234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:32.955466    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:09:33.042284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.041596    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:33.066239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.056353    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": context deadline exceeded"
Oct 30 22:09:33.108668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.097931    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:09:33.108668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.098050    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:09:33.124482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.123359    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:33.151736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.151702    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:09:33.181296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.181263    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:09:33.188983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.185447    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:09:33.188983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.188706    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:09:33.189872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.189794    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:09:33.190612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.190570    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:09:33.192621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:33.192605    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:09:34.541068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.541030    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"]
Oct 30 22:09:34.548228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.548193    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.593772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.593739    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4fn\" (UniqueName: \"kubernetes.io/projected/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-kube-api-access-2w4fn\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.593955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.593868    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.594090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.594075    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.604626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.604595    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"]
Oct 30 22:09:34.629262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.628563    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:09:34.629262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.628595    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:09:34.629262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.628603    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:09:34.629262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.628700    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:34.696290 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.696252    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4fn\" (UniqueName: \"kubernetes.io/projected/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-kube-api-access-2w4fn\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.696446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.696385    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.696675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.696646    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.697295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.697228    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.698963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.698941    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.761107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.761072    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4fn\" (UniqueName: \"kubernetes.io/projected/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-kube-api-access-2w4fn\") pod \"maas-api-revoke-sub-42ba3954-bls6x\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:34.863189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:34.863152    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:35.255998 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:35.255970    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fd6fd0_f5e8_4c24_855a_25dab5bc964f.slice/crio-4fc72741271aec86a9669f6ce1c47afca7de7812338517fee7843ae842c3800e WatchSource:0}: Error finding container 4fc72741271aec86a9669f6ce1c47afca7de7812338517fee7843ae842c3800e: Status 404 returned error can't find the container with id 4fc72741271aec86a9669f6ce1c47afca7de7812338517fee7843ae842c3800e
Oct 30 22:09:35.257004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:35.256975    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"]
Oct 30 22:09:35.974996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:35.974910    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x" event={"ID":"55fd6fd0-f5e8-4c24-855a-25dab5bc964f","Type":"ContainerStarted","Data":"512a60019c0f724febd6aa0a8fe9b8d64800a7710c2ff414b1d087dd39bb8c48"}
Oct 30 22:09:35.974996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:35.974967    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x" event={"ID":"55fd6fd0-f5e8-4c24-855a-25dab5bc964f","Type":"ContainerStarted","Data":"4fc72741271aec86a9669f6ce1c47afca7de7812338517fee7843ae842c3800e"}
Oct 30 22:09:36.098999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.098946    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x" podStartSLOduration=2.098931582 podStartE2EDuration="2.098931582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:36.097332206 +0000 UTC m=+1187.621474990" watchObservedRunningTime="2026-10-30 22:09:36.098931582 +0000 UTC m=+1187.623074340"
Oct 30 22:09:36.207740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.207704    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"]
Oct 30 22:09:36.212526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.212501    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.268879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.268799    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"]
Oct 30 22:09:36.320019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.319979    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcbt\" (UniqueName: \"kubernetes.io/projected/ba24195f-e4a5-44dc-993c-bc5de84df48c-kube-api-access-zhcbt\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.320185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.320163    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.320318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.320221    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.421887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.421849    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcbt\" (UniqueName: \"kubernetes.io/projected/ba24195f-e4a5-44dc-993c-bc5de84df48c-kube-api-access-zhcbt\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.422079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.421992    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.422079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.422034    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.422873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.422803    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.424747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.424722    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.485644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.485617    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcbt\" (UniqueName: \"kubernetes.io/projected/ba24195f-e4a5-44dc-993c-bc5de84df48c-kube-api-access-zhcbt\") pod \"maas-api-revoke-sub-c4f5a8ba-2hnw5\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.532146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.532075    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:36.715574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.715541    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:36.923374 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:36.923343    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba24195f_e4a5_44dc_993c_bc5de84df48c.slice/crio-07d66b1294d71e578edb4d17e22b5f434dce698fa1c451b7f8e44d0e59d4ade0 WatchSource:0}: Error finding container 07d66b1294d71e578edb4d17e22b5f434dce698fa1c451b7f8e44d0e59d4ade0: Status 404 returned error can't find the container with id 07d66b1294d71e578edb4d17e22b5f434dce698fa1c451b7f8e44d0e59d4ade0
Oct 30 22:09:36.923817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.923779    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"]
Oct 30 22:09:36.984974 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.984946    5031 generic.go:356] "Generic (PLEG): container finished" podID="55fd6fd0-f5e8-4c24-855a-25dab5bc964f" containerID="512a60019c0f724febd6aa0a8fe9b8d64800a7710c2ff414b1d087dd39bb8c48" exitCode=0
Oct 30 22:09:36.985359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.985027    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x" event={"ID":"55fd6fd0-f5e8-4c24-855a-25dab5bc964f","Type":"ContainerDied","Data":"512a60019c0f724febd6aa0a8fe9b8d64800a7710c2ff414b1d087dd39bb8c48"}
Oct 30 22:09:36.986527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:36.986506    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5" event={"ID":"ba24195f-e4a5-44dc-993c-bc5de84df48c","Type":"ContainerStarted","Data":"07d66b1294d71e578edb4d17e22b5f434dce698fa1c451b7f8e44d0e59d4ade0"}
Oct 30 22:09:37.997378 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:37.997339    5031 generic.go:356] "Generic (PLEG): container finished" podID="ba24195f-e4a5-44dc-993c-bc5de84df48c" containerID="69e03dcc85bff1e6b08063c3a5173427e564a6797abffe8fae9f512d3b00ef36" exitCode=0
Oct 30 22:09:37.997827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:37.997419    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5" event={"ID":"ba24195f-e4a5-44dc-993c-bc5de84df48c","Type":"ContainerDied","Data":"69e03dcc85bff1e6b08063c3a5173427e564a6797abffe8fae9f512d3b00ef36"}
Oct 30 22:09:38.226363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.226327    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:38.364852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.364814    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-xks-ca\") pod \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") "
Oct 30 22:09:38.364852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.364861    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-openshift-ca\") pod \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") "
Oct 30 22:09:38.365189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.365167    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-kube-api-access-2w4fn\" (UniqueName: \"kubernetes.io/projected/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-kube-api-access-2w4fn\") pod \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\" (UID: \"55fd6fd0-f5e8-4c24-855a-25dab5bc964f\") "
Oct 30 22:09:38.365331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.365288    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-openshift-ca" pod "55fd6fd0-f5e8-4c24-855a-25dab5bc964f" (UID: "55fd6fd0-f5e8-4c24-855a-25dab5bc964f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:38.366088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.366063    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:38.367439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.367412    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-xks-ca" pod "55fd6fd0-f5e8-4c24-855a-25dab5bc964f" (UID: "55fd6fd0-f5e8-4c24-855a-25dab5bc964f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:38.367547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.367515    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-kube-api-access-2w4fn" pod "55fd6fd0-f5e8-4c24-855a-25dab5bc964f" (UID: "55fd6fd0-f5e8-4c24-855a-25dab5bc964f"). InnerVolumeSpecName "kube-api-access-2w4fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:38.467855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.467759    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:38.467855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:38.467800    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w4fn\" (UniqueName: \"kubernetes.io/projected/55fd6fd0-f5e8-4c24-855a-25dab5bc964f-kube-api-access-2w4fn\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.008528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.008496    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"
Oct 30 22:09:39.008528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.008510    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x" event={"ID":"55fd6fd0-f5e8-4c24-855a-25dab5bc964f","Type":"ContainerDied","Data":"4fc72741271aec86a9669f6ce1c47afca7de7812338517fee7843ae842c3800e"}
Oct 30 22:09:39.009097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.008558    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fc72741271aec86a9669f6ce1c47afca7de7812338517fee7843ae842c3800e"
Oct 30 22:09:39.060656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.060574    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"]
Oct 30 22:09:39.114843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.114795    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a989cb-zt9tx"]
Oct 30 22:09:39.207738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.207706    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:39.280483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.280381    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba24195f-e4a5-44dc-993c-bc5de84df48c-kube-api-access-zhcbt\" (UniqueName: \"kubernetes.io/projected/ba24195f-e4a5-44dc-993c-bc5de84df48c-kube-api-access-zhcbt\") pod \"ba24195f-e4a5-44dc-993c-bc5de84df48c\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") "
Oct 30 22:09:39.280483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.280449    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-openshift-ca\") pod \"ba24195f-e4a5-44dc-993c-bc5de84df48c\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") "
Oct 30 22:09:39.280749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.280542    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-xks-ca\") pod \"ba24195f-e4a5-44dc-993c-bc5de84df48c\" (UID: \"ba24195f-e4a5-44dc-993c-bc5de84df48c\") "
Oct 30 22:09:39.280882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.280854    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-openshift-ca" pod "ba24195f-e4a5-44dc-993c-bc5de84df48c" (UID: "ba24195f-e4a5-44dc-993c-bc5de84df48c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:39.281101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.281073    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.282828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.282797    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-xks-ca" pod "ba24195f-e4a5-44dc-993c-bc5de84df48c" (UID: "ba24195f-e4a5-44dc-993c-bc5de84df48c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:39.282952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.282936    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba24195f-e4a5-44dc-993c-bc5de84df48c-kube-api-access-zhcbt" pod "ba24195f-e4a5-44dc-993c-bc5de84df48c" (UID: "ba24195f-e4a5-44dc-993c-bc5de84df48c"). InnerVolumeSpecName "kube-api-access-zhcbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:39.382765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.382734    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba24195f-e4a5-44dc-993c-bc5de84df48c-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.382765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:39.382763    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcbt\" (UniqueName: \"kubernetes.io/projected/ba24195f-e4a5-44dc-993c-bc5de84df48c-kube-api-access-zhcbt\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:40.021158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:40.021126    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"
Oct 30 22:09:40.021757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:40.021154    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5" event={"ID":"ba24195f-e4a5-44dc-993c-bc5de84df48c","Type":"ContainerDied","Data":"07d66b1294d71e578edb4d17e22b5f434dce698fa1c451b7f8e44d0e59d4ade0"}
Oct 30 22:09:40.021757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:40.021228    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d66b1294d71e578edb4d17e22b5f434dce698fa1c451b7f8e44d0e59d4ade0"
Oct 30 22:09:40.649473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:40.649435    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4994728-c514-4fbc-a183-61eb70d1648c" path="/var/lib/kubelet/pods/d4994728-c514-4fbc-a183-61eb70d1648c/volumes"
Oct 30 22:09:41.818422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:41.818384    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:45.094058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.094023    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"]
Oct 30 22:09:45.099060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.099033    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f0739f-xxz7x"]
Oct 30 22:09:45.533066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.532974    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:45.839170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.839084    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"]
Oct 30 22:09:45.840246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.840224    5031 cpu_manager.go:414] "containerMap: removing container" podUID="55fd6fd0-f5e8-4c24-855a-25dab5bc964f" containerName="revoke-keys"
Oct 30 22:09:45.840302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.840249    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fd6fd0-f5e8-4c24-855a-25dab5bc964f" containerName="revoke-keys"
Oct 30 22:09:45.840335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.840311    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ba24195f-e4a5-44dc-993c-bc5de84df48c" containerName="revoke-keys"
Oct 30 22:09:45.840335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.840318    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba24195f-e4a5-44dc-993c-bc5de84df48c" containerName="revoke-keys"
Oct 30 22:09:45.840491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.840479    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba24195f-e4a5-44dc-993c-bc5de84df48c" containerName="revoke-keys"
Oct 30 22:09:45.840527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.840504    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fd6fd0-f5e8-4c24-855a-25dab5bc964f" containerName="revoke-keys"
Oct 30 22:09:45.846162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.846137    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:45.901351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.901316    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"]
Oct 30 22:09:45.946087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.946046    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:45.946087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.946069    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:09:45.946398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.946102    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:09:45.951474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.951450    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:09:45.976116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.976084    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:45.976297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.976219    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:45.976373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:45.976356    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdlg\" (UniqueName: \"kubernetes.io/projected/eb42d627-3e1b-4e36-80f4-a9402d191662-kube-api-access-hfdlg\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.078299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.078261    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdlg\" (UniqueName: \"kubernetes.io/projected/eb42d627-3e1b-4e36-80f4-a9402d191662-kube-api-access-hfdlg\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.078525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.078507    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.078618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.078597    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.079182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.079156    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.081288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.081257    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.094182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.094162    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"]
Oct 30 22:09:46.098971 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.098951    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.152182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.152152    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdlg\" (UniqueName: \"kubernetes.io/projected/eb42d627-3e1b-4e36-80f4-a9402d191662-kube-api-access-hfdlg\") pod \"maas-api-revoke-sub-64fa8f78-pdjm9\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.161739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.161707    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:46.167650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.166792    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"]
Oct 30 22:09:46.180478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.180428    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59h9\" (UniqueName: \"kubernetes.io/projected/9e4b885a-a7c9-4241-9962-74330b443379-kube-api-access-j59h9\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.180594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.180518    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.180594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.180582    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.281899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.281858    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.282246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.282228    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59h9\" (UniqueName: \"kubernetes.io/projected/9e4b885a-a7c9-4241-9962-74330b443379-kube-api-access-j59h9\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.282325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.282303    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.284323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.284280    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.285157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.285131    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.357704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.357627    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59h9\" (UniqueName: \"kubernetes.io/projected/9e4b885a-a7c9-4241-9962-74330b443379-kube-api-access-j59h9\") pod \"maas-api-revoke-sub-77f1e9b7-zsgwb\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.414695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.414657    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:46.572455 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:46.572418    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb42d627_3e1b_4e36_80f4_a9402d191662.slice/crio-fa80c59c54ec62964d1c2d5519736ab89c9b529d80b51f7c50c0f30408899358 WatchSource:0}: Error finding container fa80c59c54ec62964d1c2d5519736ab89c9b529d80b51f7c50c0f30408899358: Status 404 returned error can't find the container with id fa80c59c54ec62964d1c2d5519736ab89c9b529d80b51f7c50c0f30408899358
Oct 30 22:09:46.574722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.574692    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"]
Oct 30 22:09:46.647370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.647338    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a42928-be4d-41a9-9060-709d1fb76621" path="/var/lib/kubelet/pods/44a42928-be4d-41a9-9060-709d1fb76621/volumes"
Oct 30 22:09:46.801836 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:46.801809    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4b885a_a7c9_4241_9962_74330b443379.slice/crio-ceffddc9d636c8514ab81439540c06c879f353f891e2adfd825c3ac3d4f6e084 WatchSource:0}: Error finding container ceffddc9d636c8514ab81439540c06c879f353f891e2adfd825c3ac3d4f6e084: Status 404 returned error can't find the container with id ceffddc9d636c8514ab81439540c06c879f353f891e2adfd825c3ac3d4f6e084
Oct 30 22:09:46.803270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:46.803249    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"]
Oct 30 22:09:47.087285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.087187    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb" event={"ID":"9e4b885a-a7c9-4241-9962-74330b443379","Type":"ContainerStarted","Data":"ceffddc9d636c8514ab81439540c06c879f353f891e2adfd825c3ac3d4f6e084"}
Oct 30 22:09:47.099708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.099669    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9" event={"ID":"eb42d627-3e1b-4e36-80f4-a9402d191662","Type":"ContainerStarted","Data":"0f7f48145bffac39f7b4c0d47112c9df80f3a3c4ecf141c0548779bfdf8d9036"}
Oct 30 22:09:47.100182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.099737    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9" event={"ID":"eb42d627-3e1b-4e36-80f4-a9402d191662","Type":"ContainerStarted","Data":"fa80c59c54ec62964d1c2d5519736ab89c9b529d80b51f7c50c0f30408899358"}
Oct 30 22:09:47.156888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.156853    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:47.235794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.235714    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9" podStartSLOduration=2.235692537 podStartE2EDuration="2.235692537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:47.225041858 +0000 UTC m=+1198.749184622" watchObservedRunningTime="2026-10-30 22:09:47.235692537 +0000 UTC m=+1198.759835302"
Oct 30 22:09:47.286727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.286627    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"]
Oct 30 22:09:47.292073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.292047    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.381976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.381934    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"]
Oct 30 22:09:47.400708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.400676    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.400896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.400716    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.400896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.400789    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.401030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.400949    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.401030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.400993    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd97b\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-kube-api-access-cd97b\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.401138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.401115    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1d206-820a-41f9-85e0-da237ca641ff-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.401221 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.401191    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.401330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.401313    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.401465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.401452    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1d206-820a-41f9-85e0-da237ca641ff-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.502565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502516    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1d206-820a-41f9-85e0-da237ca641ff-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.502793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502642    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.502793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502672    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.502793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502703    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.502793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502778    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd97b\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-kube-api-access-cd97b\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502891    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1d206-820a-41f9-85e0-da237ca641ff-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502969    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.502999    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.503307    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.503307    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.503812    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1d206-820a-41f9-85e0-da237ca641ff-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.503958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.503928    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.504017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.504000    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.505911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.505879    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.506038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.506018    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1d206-820a-41f9-85e0-da237ca641ff-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.568782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.568706    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.568782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.568706    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd97b\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-kube-api-access-cd97b\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:47.610563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:47.610534    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:48.064490 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:48.064456    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff1d206_820a_41f9_85e0_da237ca641ff.slice/crio-ad4c91c8667bfeaf6e3b4cb61cd6e0dd796e303a59653e919e357663f3ace95b WatchSource:0}: Error finding container ad4c91c8667bfeaf6e3b4cb61cd6e0dd796e303a59653e919e357663f3ace95b: Status 404 returned error can't find the container with id ad4c91c8667bfeaf6e3b4cb61cd6e0dd796e303a59653e919e357663f3ace95b
Oct 30 22:09:48.065319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.065290    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"]
Oct 30 22:09:48.067422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.067392    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:48.067490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.067458    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:48.067526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.067490    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:48.112069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.112034    5031 generic.go:356] "Generic (PLEG): container finished" podID="9e4b885a-a7c9-4241-9962-74330b443379" containerID="d62bbf85b30151de2d553ce032d0905235efdae098f0159f70ade566aeb498fa" exitCode=0
Oct 30 22:09:48.112458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.112125    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb" event={"ID":"9e4b885a-a7c9-4241-9962-74330b443379","Type":"ContainerDied","Data":"d62bbf85b30151de2d553ce032d0905235efdae098f0159f70ade566aeb498fa"}
Oct 30 22:09:48.114623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.114598    5031 generic.go:356] "Generic (PLEG): container finished" podID="eb42d627-3e1b-4e36-80f4-a9402d191662" containerID="0f7f48145bffac39f7b4c0d47112c9df80f3a3c4ecf141c0548779bfdf8d9036" exitCode=0
Oct 30 22:09:48.114752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.114629    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9" event={"ID":"eb42d627-3e1b-4e36-80f4-a9402d191662","Type":"ContainerDied","Data":"0f7f48145bffac39f7b4c0d47112c9df80f3a3c4ecf141c0548779bfdf8d9036"}
Oct 30 22:09:48.116383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.116361    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" event={"ID":"aff1d206-820a-41f9-85e0-da237ca641ff","Type":"ContainerStarted","Data":"ad4c91c8667bfeaf6e3b4cb61cd6e0dd796e303a59653e919e357663f3ace95b"}
Oct 30 22:09:48.341671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.341623    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"]
Oct 30 22:09:48.346666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.346641    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428144    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428195    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfwzd\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-kube-api-access-nfwzd\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428291    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428381    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428440    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3357487-cf4c-469f-b600-d02243d486e2-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428499    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428652    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3357487-cf4c-469f-b600-d02243d486e2-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428698    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.428878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.428816    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.435303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.431401    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"]
Oct 30 22:09:48.529833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.529798    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530089    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfwzd\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-kube-api-access-nfwzd\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530195    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530299    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530349    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530363    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3357487-cf4c-469f-b600-d02243d486e2-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530449    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530471    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530601    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3357487-cf4c-469f-b600-d02243d486e2-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530640    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.530730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.530714    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.531079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.531052    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.531197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.531169    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.531506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.531483    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3357487-cf4c-469f-b600-d02243d486e2-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.533016 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.532986    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.533724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.533702    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3357487-cf4c-469f-b600-d02243d486e2-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.594765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.594727    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.594924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.594823    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfwzd\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-kube-api-access-nfwzd\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.662448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.662418    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:48.869479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.869439    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"]
Oct 30 22:09:48.876478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.876454    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952405    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952524    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5x7j\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-kube-api-access-z5x7j\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952573    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952606    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952645    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952687    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952920    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952950    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.954420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.952998    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:48.958220 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.958178    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"]
Oct 30 22:09:48.976231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:48.976185    5031 scope.go:122] "RemoveContainer" containerID="1080ebfb0f7e200299a0eb953129132c62117c1f0b7db8fcb5972ff7722e9cec"
Oct 30 22:09:49.055606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.055558    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.055766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.055628    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.055766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.055676    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.055766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.055755    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.056024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.055886    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5x7j\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-kube-api-access-z5x7j\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.056024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.055938    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.056024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.055982    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.056180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.056030    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.056180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.056079    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.056853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.056823    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.056979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.056930    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.057230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.057186    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.057546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.057514    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.057907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.057881    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.061025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.061004    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.072243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.072196    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.140482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.140340    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5x7j\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-kube-api-access-z5x7j\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.141160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.141129    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.151873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.151286    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" event={"ID":"aff1d206-820a-41f9-85e0-da237ca641ff","Type":"ContainerStarted","Data":"f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3"}
Oct 30 22:09:49.154915 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:49.154890    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3357487_cf4c_469f_b600_d02243d486e2.slice/crio-2f2b1ac3347b53d05839d0121c6a43eb2301450a923c1d8c89274d4d3767df82 WatchSource:0}: Error finding container 2f2b1ac3347b53d05839d0121c6a43eb2301450a923c1d8c89274d4d3767df82: Status 404 returned error can't find the container with id 2f2b1ac3347b53d05839d0121c6a43eb2301450a923c1d8c89274d4d3767df82
Oct 30 22:09:49.158368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.158328    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.158562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.158540    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.158706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.158685    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:49.160767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.158668    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"]
Oct 30 22:09:49.200187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.199943    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:49.285127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.285074    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podStartSLOduration=2.28505912 podStartE2EDuration="2.28505912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:49.278852876 +0000 UTC m=+1200.802995655" watchObservedRunningTime="2026-10-30 22:09:49.28505912 +0000 UTC m=+1200.809201884"
Oct 30 22:09:49.466289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.461683    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:09:49.469624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.469504    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:49.470611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.470582    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:09:49.571611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.571580    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:49.593629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.593594    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-openshift-ca\") pod \"9e4b885a-a7c9-4241-9962-74330b443379\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") "
Oct 30 22:09:49.593825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.593662    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-xks-ca\") pod \"9e4b885a-a7c9-4241-9962-74330b443379\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") "
Oct 30 22:09:49.593940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.593923    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4b885a-a7c9-4241-9962-74330b443379-kube-api-access-j59h9\" (UniqueName: \"kubernetes.io/projected/9e4b885a-a7c9-4241-9962-74330b443379-kube-api-access-j59h9\") pod \"9e4b885a-a7c9-4241-9962-74330b443379\" (UID: \"9e4b885a-a7c9-4241-9962-74330b443379\") "
Oct 30 22:09:49.595401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.595368    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-openshift-ca" pod "9e4b885a-a7c9-4241-9962-74330b443379" (UID: "9e4b885a-a7c9-4241-9962-74330b443379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:49.600032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.600004    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-xks-ca" pod "9e4b885a-a7c9-4241-9962-74330b443379" (UID: "9e4b885a-a7c9-4241-9962-74330b443379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:49.600522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.600501    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4b885a-a7c9-4241-9962-74330b443379-kube-api-access-j59h9" pod "9e4b885a-a7c9-4241-9962-74330b443379" (UID: "9e4b885a-a7c9-4241-9962-74330b443379"). InnerVolumeSpecName "kube-api-access-j59h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:49.650804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.650384    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:49.650804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.650495    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:09:49.650804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.650536    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:09:49.696087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.695984    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-xks-ca\") pod \"eb42d627-3e1b-4e36-80f4-a9402d191662\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") "
Oct 30 22:09:49.696087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.696076    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42d627-3e1b-4e36-80f4-a9402d191662-kube-api-access-hfdlg\" (UniqueName: \"kubernetes.io/projected/eb42d627-3e1b-4e36-80f4-a9402d191662-kube-api-access-hfdlg\") pod \"eb42d627-3e1b-4e36-80f4-a9402d191662\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") "
Oct 30 22:09:49.696697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.696675    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-openshift-ca\") pod \"eb42d627-3e1b-4e36-80f4-a9402d191662\" (UID: \"eb42d627-3e1b-4e36-80f4-a9402d191662\") "
Oct 30 22:09:49.698250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.698225    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59h9\" (UniqueName: \"kubernetes.io/projected/9e4b885a-a7c9-4241-9962-74330b443379-kube-api-access-j59h9\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:49.698349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.698254    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:49.698349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.698271    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4b885a-a7c9-4241-9962-74330b443379-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:49.698479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.698431    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-openshift-ca" pod "eb42d627-3e1b-4e36-80f4-a9402d191662" (UID: "eb42d627-3e1b-4e36-80f4-a9402d191662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:49.699476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.699450    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42d627-3e1b-4e36-80f4-a9402d191662-kube-api-access-hfdlg" pod "eb42d627-3e1b-4e36-80f4-a9402d191662" (UID: "eb42d627-3e1b-4e36-80f4-a9402d191662"). InnerVolumeSpecName "kube-api-access-hfdlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:49.700712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.700688    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-xks-ca" pod "eb42d627-3e1b-4e36-80f4-a9402d191662" (UID: "eb42d627-3e1b-4e36-80f4-a9402d191662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:49.799741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.799641    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfdlg\" (UniqueName: \"kubernetes.io/projected/eb42d627-3e1b-4e36-80f4-a9402d191662-kube-api-access-hfdlg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:49.799741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.799679    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:49.799741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:49.799696    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb42d627-3e1b-4e36-80f4-a9402d191662-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:50.001543 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:09:50.001511    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9e73e0_ea09_42b6_9c5f_c9c4c44071d8.slice/crio-2d56e28ad386ea2c843156369735d7576aade6e8b6f26597f00239e8654bc407 WatchSource:0}: Error finding container 2d56e28ad386ea2c843156369735d7576aade6e8b6f26597f00239e8654bc407: Status 404 returned error can't find the container with id 2d56e28ad386ea2c843156369735d7576aade6e8b6f26597f00239e8654bc407
Oct 30 22:09:50.002545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.002517    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"]
Oct 30 22:09:50.004454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.004419    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:50.004537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.004492    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:50.004537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.004523    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:09:50.172011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.171980    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" event={"ID":"f3357487-cf4c-469f-b600-d02243d486e2","Type":"ContainerStarted","Data":"66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d"}
Oct 30 22:09:50.172380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.172033    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" event={"ID":"f3357487-cf4c-469f-b600-d02243d486e2","Type":"ContainerStarted","Data":"2f2b1ac3347b53d05839d0121c6a43eb2301450a923c1d8c89274d4d3767df82"}
Oct 30 22:09:50.174359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.174334    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb" event={"ID":"9e4b885a-a7c9-4241-9962-74330b443379","Type":"ContainerDied","Data":"ceffddc9d636c8514ab81439540c06c879f353f891e2adfd825c3ac3d4f6e084"}
Oct 30 22:09:50.174482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.174376    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceffddc9d636c8514ab81439540c06c879f353f891e2adfd825c3ac3d4f6e084"
Oct 30 22:09:50.174482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.174420    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"
Oct 30 22:09:50.177684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.177656    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9" event={"ID":"eb42d627-3e1b-4e36-80f4-a9402d191662","Type":"ContainerDied","Data":"fa80c59c54ec62964d1c2d5519736ab89c9b529d80b51f7c50c0f30408899358"}
Oct 30 22:09:50.177798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.177708    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa80c59c54ec62964d1c2d5519736ab89c9b529d80b51f7c50c0f30408899358"
Oct 30 22:09:50.177798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.177765    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"
Oct 30 22:09:50.181555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.181477    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" event={"ID":"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8","Type":"ContainerStarted","Data":"2d56e28ad386ea2c843156369735d7576aade6e8b6f26597f00239e8654bc407"}
Oct 30 22:09:50.341297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.340173    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podStartSLOduration=2.340150739 podStartE2EDuration="2.340150739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:50.309999267 +0000 UTC m=+1201.834142032" watchObservedRunningTime="2026-10-30 22:09:50.340150739 +0000 UTC m=+1201.864293512"
Oct 30 22:09:50.596903 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:09:50.596860    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:09:50.611983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.611492    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:09:50.611983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.611567    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:09:50.668254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.667245    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:09:50.668254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.667299    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:09:50.675581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:50.675057    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:51.200270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.200193    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" event={"ID":"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8","Type":"ContainerStarted","Data":"19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905"}
Oct 30 22:09:51.204678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.204653    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:51.205944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.205917    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:09:51.206069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.205973    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:09:51.361007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.360164    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podStartSLOduration=3.360138787 podStartE2EDuration="3.360138787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:51.33977035 +0000 UTC m=+1202.863913127" watchObservedRunningTime="2026-10-30 22:09:51.360138787 +0000 UTC m=+1202.884281552"
Oct 30 22:09:51.626235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.612685    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:09:51.626235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.612757    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:09:51.668235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.667727    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:09:51.668235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:51.667808    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:09:52.226237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:52.222354    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:09:52.226237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:52.222422    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:09:52.625236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:52.615676    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:09:52.625236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:52.615741    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:09:52.666236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:52.665543    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:09:52.666236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:52.665602    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:09:52.869229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:52.868409    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:53.264400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.264362    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:09:53.265368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.264954    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:09:53.636732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.636689    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:09:53.637074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.637021    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:09:53.671057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.671023    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:09:53.671382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.671337    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:09:53.842238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.824674    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:53.846523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:53.843651    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:54.035539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.015315    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Oct 30 22:09:54.119320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.117481    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:09:54.245678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.211312    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:09:54.245678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.211372    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:09:54.264068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.262976    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:54.265279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.264764    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:09:54.684429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.683331    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:09:54.684429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.683411    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:09:54.684429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.684252    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:09:54.685787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:54.685708    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:09:55.226671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:55.226632    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.233598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:55.233549    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:09:55.611675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:55.611636    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.612364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:55.612330    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:09:55.664676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:55.664636    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.664962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:55.664930    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:09:56.069057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.068954    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"]
Oct 30 22:09:56.141234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.140753    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f04383-592z9"]
Oct 30 22:09:56.205956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.205924    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:09:56.206245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.206190    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:09:56.633602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.633558    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:56.636326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.636302    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:56.656085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.656052    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde" path="/var/lib/kubelet/pods/7924d7a1-c1ba-41f2-aa1f-1b7adcef9cde/volumes"
Oct 30 22:09:56.656756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.656734    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:09:56.663222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.663184    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:09:56.663351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:56.663267    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:09:57.206230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:57.204724    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:09:57.206230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:57.204791    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:09:57.668523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:57.668489    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:57.668960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:57.668888    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:57.669765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:57.669741    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:09:58.070959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:58.070826    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"]
Oct 30 22:09:58.076806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:58.076770    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6488f356-87gpb"]
Oct 30 22:09:58.348298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:58.346695    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:58.348298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:58.347904    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:58.350226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:58.349495    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:09:58.668558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:58.668471    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9" path="/var/lib/kubelet/pods/70e48ce4-ecf0-48d6-9c7d-fd43e5ce71e9/volumes"
Oct 30 22:09:58.776792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:09:58.776757    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:03.433847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.433809    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"]
Oct 30 22:10:03.434605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.434589    5031 cpu_manager.go:414] "containerMap: removing container" podUID="9e4b885a-a7c9-4241-9962-74330b443379" containerName="revoke-keys"
Oct 30 22:10:03.434658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.434607    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4b885a-a7c9-4241-9962-74330b443379" containerName="revoke-keys"
Oct 30 22:10:03.434658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.434636    5031 cpu_manager.go:414] "containerMap: removing container" podUID="eb42d627-3e1b-4e36-80f4-a9402d191662" containerName="revoke-keys"
Oct 30 22:10:03.434658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.434641    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb42d627-3e1b-4e36-80f4-a9402d191662" containerName="revoke-keys"
Oct 30 22:10:03.434790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.434781    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4b885a-a7c9-4241-9962-74330b443379" containerName="revoke-keys"
Oct 30 22:10:03.434822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.434798    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb42d627-3e1b-4e36-80f4-a9402d191662" containerName="revoke-keys"
Oct 30 22:10:03.439526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.439502    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543136    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543325    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543465    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543605    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543651    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543792    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543828    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.543861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543855    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cchls\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-kube-api-access-cchls\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.544035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.543908    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.552743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.552707    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"]
Oct 30 22:10:03.645117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645083    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645122    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645144    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cchls\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-kube-api-access-cchls\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645267    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645365    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645462    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645555    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645577    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645628    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645696    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645739    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.645814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.645742    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.646136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.646115    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.646466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.646443    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.647944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.647922    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.649004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.648985    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.718688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.718612    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cchls\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-kube-api-access-cchls\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.721433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.721397    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.759523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.759492    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:03.926241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:03.919589    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:04.062780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.062703    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"]
Oct 30 22:10:04.067632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.067600    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ef94f-cbxq7"]
Oct 30 22:10:04.231745 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:04.231700    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5c548c_8eca_485e_a5c8_22affe05b1f7.slice/crio-5d468ff0a48dbaa760561553ab438840f114e6164ec7d516b1d541d35648bfc8 WatchSource:0}: Error finding container 5d468ff0a48dbaa760561553ab438840f114e6164ec7d516b1d541d35648bfc8: Status 404 returned error can't find the container with id 5d468ff0a48dbaa760561553ab438840f114e6164ec7d516b1d541d35648bfc8
Oct 30 22:10:04.236553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.233529    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"]
Oct 30 22:10:04.239587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.239545    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:04.239693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.239638    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:04.239693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.239678    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:04.653964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.653920    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39ad791-00d3-4538-8b2a-420e184c5fdd" path="/var/lib/kubelet/pods/d39ad791-00d3-4538-8b2a-420e184c5fdd/volumes"
Oct 30 22:10:04.692574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.692517    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" event={"ID":"1f5c548c-8eca-485e-a5c8-22affe05b1f7","Type":"ContainerStarted","Data":"88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86"}
Oct 30 22:10:04.692876 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.692848    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" event={"ID":"1f5c548c-8eca-485e-a5c8-22affe05b1f7","Type":"ContainerStarted","Data":"5d468ff0a48dbaa760561553ab438840f114e6164ec7d516b1d541d35648bfc8"}
Oct 30 22:10:04.760111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:04.760065    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:05.764404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:05.764251    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:05.764404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:05.764328    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:06.791235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:06.789879    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:06.791235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:06.789939    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:06.791845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:06.791324    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.083298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.079311    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.156077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.088792    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:10:07.165378 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.164521    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.212235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.206807    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.212235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.207254    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:10:07.212235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.207585    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:10:07.212235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.207871    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:10:07.212235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.208135    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:10:07.527238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.510364    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.564157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.562599    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.768237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.760647    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:07.768237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.760710    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:07.802302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.801330    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:07.802302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.801556    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:10:07.927238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:07.926426    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podStartSLOduration=4.926403079 podStartE2EDuration="4.926403079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:04.824461605 +0000 UTC m=+1216.348604370" watchObservedRunningTime="2026-10-30 22:10:07.926403079 +0000 UTC m=+1219.450545845"
Oct 30 22:10:08.036403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.036333    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:08.105654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.105608    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:10:08.106370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.106345    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:10:08.354331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.354282    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:08.357501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.355337    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:10:08.357743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.355367    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:10:08.357875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.355390    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:10:08.357978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.355406    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:10:08.358071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.355430    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:10:08.358158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.355449    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:08.378596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.362101    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:10:08.378596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.362172    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:10:08.378596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.362232    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:10:08.378596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.369510    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:10:08.378596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.369567    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:10:08.378596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.369605    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:10:08.823307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.810403    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:08.849727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.845457    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:08.849727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:08.845536    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:09.089295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.065294    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:10:09.154247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.106547    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:09.347365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.347329    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:10:09.347653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.347633    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"]
Oct 30 22:10:09.356368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.356333    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"]
Oct 30 22:10:09.372316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.356790    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" containerID="cri-o://6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d" gracePeriod=30
Oct 30 22:10:09.372736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.357809    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" containerID="cri-o://9dfd65acf94c4e4eca24eceed2632bd4c8cd8173e90ac798930886a415145270" gracePeriod=30
Oct 30 22:10:09.380413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.359467    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:09.404846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.376653    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:10:09.404846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.380874    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:09.404846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.380908    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:09.404846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.380934    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:10:09.404846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.382425    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:09.609410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.586801    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:10:09.638237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.614327    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:09.782020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.781977    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"]
Oct 30 22:10:09.783047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.783006    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy" containerID="cri-o://bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0" gracePeriod=30
Oct 30 22:10:09.809389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.808516    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:10:09.833449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.826016    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:10:09.976336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.960653    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:10.013257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:09.960718    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:10.091392 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.091345    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:10.108073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.107955    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:10:10.108384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.108368    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:10:10.137588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.137490    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:10:10.154235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.140921    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:10.164524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.158112    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:10:10.164524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.159585    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:10:10.164524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.161674    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"]
Oct 30 22:10:10.189235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.184452    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:10:10.189235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.184488    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"]
Oct 30 22:10:10.189235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.185437    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.194696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.192780    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:10:10.202228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.200497    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:10:10.245040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.245007    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:10.246363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.246028    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:10.246816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.246778    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:10:10.248389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.248328    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:10.294128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.294088    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.302826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.302790    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rt6w\" (UniqueName: \"kubernetes.io/projected/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-kube-api-access-5rt6w\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.355235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.345113    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.360283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.358885    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:10.452989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.452503    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.452989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.452609    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rt6w\" (UniqueName: \"kubernetes.io/projected/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-kube-api-access-5rt6w\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.452989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.452706    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.453717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.453620    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.481538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.481502    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.566131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.566043    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rt6w\" (UniqueName: \"kubernetes.io/projected/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-kube-api-access-5rt6w\") pod \"maas-api-revoke-sub-9f2a9195-tmltq\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.640238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.631460    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:10.769843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.764901    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:10.769843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:10.764970    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:11.171435 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.171386    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"]
Oct 30 22:10:11.289659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.289552    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"]
Oct 30 22:10:11.297422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.297391    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.357799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.357764    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"]
Oct 30 22:10:11.380120 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.380002    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.380281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.380131    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.380281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.380267    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz885\" (UniqueName: \"kubernetes.io/projected/4c8c4687-ba3b-4509-98e6-907006e836c1-kube-api-access-cz885\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.489080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.485970    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.489080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.486062    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.489080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.486134    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz885\" (UniqueName: \"kubernetes.io/projected/4c8c4687-ba3b-4509-98e6-907006e836c1-kube-api-access-cz885\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.489080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.487040    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.491187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.491161    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.551231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.550999    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz885\" (UniqueName: \"kubernetes.io/projected/4c8c4687-ba3b-4509-98e6-907006e836c1-kube-api-access-cz885\") pod \"maas-api-revoke-sub-946a92d0-mdxlq\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.643292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.642469    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:11.766370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.766336    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:11.766539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.766396    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:11.784258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.782479    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq" event={"ID":"7d5b4e48-c6ff-4025-9193-9f09f4488bc9","Type":"ContainerStarted","Data":"4f5c4f6682ed600437ea719d938edce76ee831dba7e8ac9defd8cc3395823d53"}
Oct 30 22:10:11.784258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.782575    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq" event={"ID":"7d5b4e48-c6ff-4025-9193-9f09f4488bc9","Type":"ContainerStarted","Data":"faed5818c4f9ec58833a09806659c8f1e426804aa9c515354635f6146768e134"}
Oct 30 22:10:11.877147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.876052    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:11.877147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:11.876113    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:12.091806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.091766    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"]
Oct 30 22:10:12.091942 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:12.091787    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8c4687_ba3b_4509_98e6_907006e836c1.slice/crio-0a0b9c97c96370f0bd48354bd9745d9fdde445f898c97c1a52079b636ee33224 WatchSource:0}: Error finding container 0a0b9c97c96370f0bd48354bd9745d9fdde445f898c97c1a52079b636ee33224: Status 404 returned error can't find the container with id 0a0b9c97c96370f0bd48354bd9745d9fdde445f898c97c1a52079b636ee33224
Oct 30 22:10:12.391546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.391485    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq" podStartSLOduration=4.391464528 podStartE2EDuration="4.391464528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:12.388348259 +0000 UTC m=+1223.912491046" watchObservedRunningTime="2026-10-30 22:10:12.391464528 +0000 UTC m=+1223.915607292"
Oct 30 22:10:12.760920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.760825    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:12.760920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.760886    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:12.793636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.793600    5031 generic.go:356] "Generic (PLEG): container finished" podID="7d5b4e48-c6ff-4025-9193-9f09f4488bc9" containerID="4f5c4f6682ed600437ea719d938edce76ee831dba7e8ac9defd8cc3395823d53" exitCode=0
Oct 30 22:10:12.793806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.793739    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq" event={"ID":"7d5b4e48-c6ff-4025-9193-9f09f4488bc9","Type":"ContainerDied","Data":"4f5c4f6682ed600437ea719d938edce76ee831dba7e8ac9defd8cc3395823d53"}
Oct 30 22:10:12.795891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.795864    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq" event={"ID":"4c8c4687-ba3b-4509-98e6-907006e836c1","Type":"ContainerStarted","Data":"a3742fe9b82851a4c98291f89c694899b2b5d7daa0fdd451af8ad1d66aac9214"}
Oct 30 22:10:12.796019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:12.795917    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq" event={"ID":"4c8c4687-ba3b-4509-98e6-907006e836c1","Type":"ContainerStarted","Data":"0a0b9c97c96370f0bd48354bd9745d9fdde445f898c97c1a52079b636ee33224"}
Oct 30 22:10:13.033422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.033307    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq" podStartSLOduration=2.033292689 podStartE2EDuration="2.033292689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:13.032373805 +0000 UTC m=+1224.556516570" watchObservedRunningTime="2026-10-30 22:10:13.033292689 +0000 UTC m=+1224.557435452"
Oct 30 22:10:13.620920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.620883    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"]
Oct 30 22:10:13.625925 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.625895    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.685940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.685903    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"]
Oct 30 22:10:13.749274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.749226    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.749664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.749636    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.749762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.749739    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfq2f\" (UniqueName: \"kubernetes.io/projected/ea3a6021-8bbf-480a-ace7-98426d786963-kube-api-access-vfq2f\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.760686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.760648    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:10:13.760815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.760729    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:10:13.806865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.806833    5031 generic.go:356] "Generic (PLEG): container finished" podID="4c8c4687-ba3b-4509-98e6-907006e836c1" containerID="a3742fe9b82851a4c98291f89c694899b2b5d7daa0fdd451af8ad1d66aac9214" exitCode=0
Oct 30 22:10:13.807060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.806924    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq" event={"ID":"4c8c4687-ba3b-4509-98e6-907006e836c1","Type":"ContainerDied","Data":"a3742fe9b82851a4c98291f89c694899b2b5d7daa0fdd451af8ad1d66aac9214"}
Oct 30 22:10:13.851397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.851363    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.851580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.851548    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.851669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.851582    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfq2f\" (UniqueName: \"kubernetes.io/projected/ea3a6021-8bbf-480a-ace7-98426d786963-kube-api-access-vfq2f\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.852466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.852438    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.855657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.855599    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.924529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.922452    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfq2f\" (UniqueName: \"kubernetes.io/projected/ea3a6021-8bbf-480a-ace7-98426d786963-kube-api-access-vfq2f\") pod \"maas-api-revoke-sub-e5197cc9-8hn4n\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:13.942733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:13.942689    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:14.042625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.042514    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:14.071369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.071332    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"]
Oct 30 22:10:14.076793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.076753    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b15fd75-fnlqv"]
Oct 30 22:10:14.170291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.170163    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-openshift-ca\") pod \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") "
Oct 30 22:10:14.170457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.170357    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-kube-api-access-5rt6w\" (UniqueName: \"kubernetes.io/projected/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-kube-api-access-5rt6w\") pod \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") "
Oct 30 22:10:14.170592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.170574    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-xks-ca\") pod \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\" (UID: \"7d5b4e48-c6ff-4025-9193-9f09f4488bc9\") "
Oct 30 22:10:14.171881 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.171847    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-openshift-ca" pod "7d5b4e48-c6ff-4025-9193-9f09f4488bc9" (UID: "7d5b4e48-c6ff-4025-9193-9f09f4488bc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:14.175009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.174984    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-kube-api-access-5rt6w" pod "7d5b4e48-c6ff-4025-9193-9f09f4488bc9" (UID: "7d5b4e48-c6ff-4025-9193-9f09f4488bc9"). InnerVolumeSpecName "kube-api-access-5rt6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:14.175838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.175806    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-xks-ca" pod "7d5b4e48-c6ff-4025-9193-9f09f4488bc9" (UID: "7d5b4e48-c6ff-4025-9193-9f09f4488bc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:14.273031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.272994    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rt6w\" (UniqueName: \"kubernetes.io/projected/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-kube-api-access-5rt6w\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:14.273031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.273028    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:14.273031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.273040    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b4e48-c6ff-4025-9193-9f09f4488bc9-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:14.373812 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:14.373776    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3a6021_8bbf_480a_ace7_98426d786963.slice/crio-f7e290e07a77ae0f5a06d50e3668ffb373e8df27faca8d36a378400c3a2bb12f WatchSource:0}: Error finding container f7e290e07a77ae0f5a06d50e3668ffb373e8df27faca8d36a378400c3a2bb12f: Status 404 returned error can't find the container with id f7e290e07a77ae0f5a06d50e3668ffb373e8df27faca8d36a378400c3a2bb12f
Oct 30 22:10:14.378280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.378248    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"]
Oct 30 22:10:14.648439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.648396    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65d9e8d-8440-4322-ab04-cfe33c48f1f5" path="/var/lib/kubelet/pods/f65d9e8d-8440-4322-ab04-cfe33c48f1f5/volumes"
Oct 30 22:10:14.765584 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.765532    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:14.766043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.765999    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:14.766738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.766717    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:10:14.824575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.824549    5031 generic.go:356] "Generic (PLEG): container finished" podID="cae72439-1274-4abd-8182-6f918935d93f" containerID="9dfd65acf94c4e4eca24eceed2632bd4c8cd8173e90ac798930886a415145270" exitCode=0
Oct 30 22:10:14.824704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.824622    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" event={"ID":"cae72439-1274-4abd-8182-6f918935d93f","Type":"ContainerDied","Data":"9dfd65acf94c4e4eca24eceed2632bd4c8cd8173e90ac798930886a415145270"}
Oct 30 22:10:14.826265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.826236    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq" event={"ID":"7d5b4e48-c6ff-4025-9193-9f09f4488bc9","Type":"ContainerDied","Data":"faed5818c4f9ec58833a09806659c8f1e426804aa9c515354635f6146768e134"}
Oct 30 22:10:14.826391 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.826274    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"
Oct 30 22:10:14.826391 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.826289    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faed5818c4f9ec58833a09806659c8f1e426804aa9c515354635f6146768e134"
Oct 30 22:10:14.828238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:14.828198    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n" event={"ID":"ea3a6021-8bbf-480a-ace7-98426d786963","Type":"ContainerStarted","Data":"f7e290e07a77ae0f5a06d50e3668ffb373e8df27faca8d36a378400c3a2bb12f"}
Oct 30 22:10:15.085246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.084699    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"]
Oct 30 22:10:15.105689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.101358    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"]
Oct 30 22:10:15.112386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.112317    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e434794e-sjltg"]
Oct 30 22:10:15.123761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.123726    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9984e673-zz2xw"]
Oct 30 22:10:15.248731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.248340    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 22:10:15.329969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.329941    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:10:15.334433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.334364    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:15.359422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.357110    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:15.386585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.386553    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:10:15.407253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.407026    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-istio-token\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-istio-token\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.407253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.407107    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-kube-api-access-4ll9g\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-kube-api-access-4ll9g\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.407253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.407236    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-credential-socket\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.413734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.413375    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-envoy\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.414392 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.414073    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-data\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.414392 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.414353    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-data" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.414969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.414624    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c671a13e-3166-4293-a989-956f2f6f2873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c671a13e-3166-4293-a989-956f2f6f2873-istiod-ca-cert\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.414969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.414943    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c671a13e-3166-4293-a989-956f2f6f2873-istiod-ca-cert" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:15.415610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.415290    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-envoy\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.415698    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-certs\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.415776    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c671a13e-3166-4293-a989-956f2f6f2873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c671a13e-3166-4293-a989-956f2f6f2873-istio-podinfo\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.415853    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-istio-token\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-istio-token\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.415889    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-socket\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.415922    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-socket\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.415968    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8c4687-ba3b-4509-98e6-907006e836c1-kube-api-access-cz885\" (UniqueName: \"kubernetes.io/projected/4c8c4687-ba3b-4509-98e6-907006e836c1-kube-api-access-cz885\") pod \"4c8c4687-ba3b-4509-98e6-907006e836c1\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.416004    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-openshift-ca\") pod \"4c8c4687-ba3b-4509-98e6-907006e836c1\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.416097    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-kube-api-access-8g9bg\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-kube-api-access-8g9bg\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.416243    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-envoy" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.416508    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-socket" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.416898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.416668    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-certs" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.417928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.417844    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae72439-1274-4abd-8182-6f918935d93f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae72439-1274-4abd-8182-6f918935d93f-istiod-ca-cert\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.417928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.417879    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cae72439-1274-4abd-8182-6f918935d93f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae72439-1274-4abd-8182-6f918935d93f-istio-podinfo\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.418443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.418140    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-socket" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.418443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.418240    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-data\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.418443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.418297    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-certs\") pod \"cae72439-1274-4abd-8182-6f918935d93f\" (UID: \"cae72439-1274-4abd-8182-6f918935d93f\") "
Oct 30 22:10:15.418443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.418337    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-xks-ca\") pod \"4c8c4687-ba3b-4509-98e6-907006e836c1\" (UID: \"4c8c4687-ba3b-4509-98e6-907006e836c1\") "
Oct 30 22:10:15.418443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.418385    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-credential-socket\") pod \"c671a13e-3166-4293-a989-956f2f6f2873\" (UID: \"c671a13e-3166-4293-a989-956f2f6f2873\") "
Oct 30 22:10:15.419313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.419288    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-certs" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.419556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.419523    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-data" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.419666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.419649    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c671a13e-3166-4293-a989-956f2f6f2873-istio-podinfo" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:15.420320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420040    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-openshift-ca" pod "4c8c4687-ba3b-4509-98e6-907006e836c1" (UID: "4c8c4687-ba3b-4509-98e6-907006e836c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:15.420410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420312    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-credential-socket" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.420831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420812    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.420901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420838    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.420901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420856    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.420901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420871    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.420901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420885    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.420901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420898    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c671a13e-3166-4293-a989-956f2f6f2873-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.421137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420914    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.421137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420927    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c671a13e-3166-4293-a989-956f2f6f2873-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.421137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420942    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c671a13e-3166-4293-a989-956f2f6f2873-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.421137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420956    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.421137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.420969    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.422443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.421497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae72439-1274-4abd-8182-6f918935d93f-istiod-ca-cert" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:15.422443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.422093    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8c4687-ba3b-4509-98e6-907006e836c1-kube-api-access-cz885" pod "4c8c4687-ba3b-4509-98e6-907006e836c1" (UID: "4c8c4687-ba3b-4509-98e6-907006e836c1"). InnerVolumeSpecName "kube-api-access-cz885". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.422443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.422229    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-xks-ca" pod "4c8c4687-ba3b-4509-98e6-907006e836c1" (UID: "4c8c4687-ba3b-4509-98e6-907006e836c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:15.423324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.423199    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cae72439-1274-4abd-8182-6f918935d93f-istio-podinfo" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:15.424041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.424009    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-kube-api-access-8g9bg" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "kube-api-access-8g9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.425168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.425140    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-istio-token" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.426276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.426249    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-credential-socket" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.429418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.429394    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-kube-api-access-4ll9g" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "kube-api-access-4ll9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.429712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.429684    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-envoy" pod "cae72439-1274-4abd-8182-6f918935d93f" (UID: "cae72439-1274-4abd-8182-6f918935d93f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.434592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.434396    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-istio-token" pod "c671a13e-3166-4293-a989-956f2f6f2873" (UID: "c671a13e-3166-4293-a989-956f2f6f2873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.524143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524031    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-podinfo\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524098    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-socket\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524145    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-envoy\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524354    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istiod-ca-cert\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524480    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-credential-socket\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524596    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-kube-api-access-5sl7k\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-kube-api-access-5sl7k\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524651    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-token\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-token\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524689    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-certs\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.524804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.524782    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-data\") pod \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\" (UID: \"e5a7728b-68ff-4812-b8ea-e7ea9385e08c\") "
Oct 30 22:10:15.525184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.525152    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istiod-ca-cert" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:15.525353 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.525286    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-socket" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.525691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.525497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-credential-socket" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.525691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.525662    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-certs" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.526004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.525788    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-data" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.526080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526066    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526085    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526100    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526113    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526126    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz885\" (UniqueName: \"kubernetes.io/projected/4c8c4687-ba3b-4509-98e6-907006e836c1-kube-api-access-cz885\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526140    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g9bg\" (UniqueName: \"kubernetes.io/projected/cae72439-1274-4abd-8182-6f918935d93f-kube-api-access-8g9bg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526155    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae72439-1274-4abd-8182-6f918935d93f-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526170    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae72439-1274-4abd-8182-6f918935d93f-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526184    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.526534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526197    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c4687-ba3b-4509-98e6-907006e836c1-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.527235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526660    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.527235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526687    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.527235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526702    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ll9g\" (UniqueName: \"kubernetes.io/projected/c671a13e-3166-4293-a989-956f2f6f2873-kube-api-access-4ll9g\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.527235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526717    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae72439-1274-4abd-8182-6f918935d93f-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.527235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.526731    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.528652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.528623    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-podinfo" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:15.529785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.529760    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-token" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.530004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.529976    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-kube-api-access-5sl7k" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "kube-api-access-5sl7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.537089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.537057    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-envoy" pod "e5a7728b-68ff-4812-b8ea-e7ea9385e08c" (UID: "e5a7728b-68ff-4812-b8ea-e7ea9385e08c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.629060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.629016    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.629060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.629064    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.629307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.629081    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.629307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.629096    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sl7k\" (UniqueName: \"kubernetes.io/projected/e5a7728b-68ff-4812-b8ea-e7ea9385e08c-kube-api-access-5sl7k\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.844508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.844453    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq" event={"ID":"4c8c4687-ba3b-4509-98e6-907006e836c1","Type":"ContainerDied","Data":"0a0b9c97c96370f0bd48354bd9745d9fdde445f898c97c1a52079b636ee33224"}
Oct 30 22:10:15.844508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.844525    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a0b9c97c96370f0bd48354bd9745d9fdde445f898c97c1a52079b636ee33224"
Oct 30 22:10:15.845280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.845259    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"
Oct 30 22:10:15.846643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.846607    5031 generic.go:356] "Generic (PLEG): container finished" podID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerID="bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0" exitCode=0
Oct 30 22:10:15.846818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.846654    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"
Oct 30 22:10:15.846818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.846786    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" event={"ID":"e5a7728b-68ff-4812-b8ea-e7ea9385e08c","Type":"ContainerDied","Data":"bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0"}
Oct 30 22:10:15.846979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.846833    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4" event={"ID":"e5a7728b-68ff-4812-b8ea-e7ea9385e08c","Type":"ContainerDied","Data":"d04ad3a2d8bab5a222015503deeba340e02451a46cf2a7f0c50b3e52a305c76f"}
Oct 30 22:10:15.846979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.846863    5031 scope.go:122] "RemoveContainer" containerID="bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0"
Oct 30 22:10:15.854310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.854283    5031 generic.go:356] "Generic (PLEG): container finished" podID="ea3a6021-8bbf-480a-ace7-98426d786963" containerID="075e36434ebef4994b1bd1fd2825d6f225be4ee028ce38b2be768c64ae6923ff" exitCode=0
Oct 30 22:10:15.854478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.854459    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n" event={"ID":"ea3a6021-8bbf-480a-ace7-98426d786963","Type":"ContainerDied","Data":"075e36434ebef4994b1bd1fd2825d6f225be4ee028ce38b2be768c64ae6923ff"}
Oct 30 22:10:15.857113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.857085    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr" event={"ID":"cae72439-1274-4abd-8182-6f918935d93f","Type":"ContainerDied","Data":"cf001b17f7bfb5dfe504556ec05511627d80963e12908f618e9a01fb7d57a8d0"}
Oct 30 22:10:15.857362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.857335    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"
Oct 30 22:10:15.861790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.861653    5031 generic.go:356] "Generic (PLEG): container finished" podID="c671a13e-3166-4293-a989-956f2f6f2873" containerID="6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d" exitCode=0
Oct 30 22:10:15.862609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.862098    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"
Oct 30 22:10:15.864653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.863870    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" event={"ID":"c671a13e-3166-4293-a989-956f2f6f2873","Type":"ContainerDied","Data":"6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d"}
Oct 30 22:10:15.864653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.863924    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" event={"ID":"c671a13e-3166-4293-a989-956f2f6f2873","Type":"ContainerDied","Data":"afa799073b95e80ea92956d7ca2290b6a42b844ba377c05228eea5333353f4dd"}
Oct 30 22:10:15.869402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.869377    5031 scope.go:122] "RemoveContainer" containerID="bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0"
Oct 30 22:10:15.869809 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:10:15.869789    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0\": container with ID starting with bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0 not found: ID does not exist" containerID="bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0"
Oct 30 22:10:15.869899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.869827    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0"} err="failed to get container status \"bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0\": rpc error: code = NotFound desc = could not find container \"bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0\": container with ID starting with bfb24b16180bf0b932553ad2b0769e0981982dc15612dd0faf74b87d85b191e0 not found: ID does not exist"
Oct 30 22:10:15.869899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.869863    5031 scope.go:122] "RemoveContainer" containerID="9dfd65acf94c4e4eca24eceed2632bd4c8cd8173e90ac798930886a415145270"
Oct 30 22:10:15.922502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.922478    5031 scope.go:122] "RemoveContainer" containerID="6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d"
Oct 30 22:10:15.937117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.937092    5031 scope.go:122] "RemoveContainer" containerID="6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d"
Oct 30 22:10:15.937550 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:10:15.937529    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d\": container with ID starting with 6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d not found: ID does not exist" containerID="6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d"
Oct 30 22:10:15.937622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:15.937561    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d"} err="failed to get container status \"6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d\": rpc error: code = NotFound desc = could not find container \"6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d\": container with ID starting with 6e6cb44fabda9b3ed51783153492961b1fe2992c26f11b9c7f56aee211e2b36d not found: ID does not exist"
Oct 30 22:10:16.186062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.185967    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:10:16.186062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.186026    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:16.214713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.214668    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"]
Oct 30 22:10:16.215556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215530    5031 cpu_manager.go:414] "containerMap: removing container" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy"
Oct 30 22:10:16.215556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215556    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215583    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4c8c4687-ba3b-4509-98e6-907006e836c1" containerName="revoke-keys"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215591    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8c4687-ba3b-4509-98e6-907006e836c1" containerName="revoke-keys"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215607    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215616    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215635    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7d5b4e48-c6ff-4025-9193-9f09f4488bc9" containerName="revoke-keys"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215641    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5b4e48-c6ff-4025-9193-9f09f4488bc9" containerName="revoke-keys"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215655    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy"
Oct 30 22:10:16.215811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215661    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy"
Oct 30 22:10:16.216167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215861    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae72439-1274-4abd-8182-6f918935d93f" containerName="istio-proxy"
Oct 30 22:10:16.216167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215879    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8c4687-ba3b-4509-98e6-907006e836c1" containerName="revoke-keys"
Oct 30 22:10:16.216167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215890    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" containerName="istio-proxy"
Oct 30 22:10:16.216167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215901    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5b4e48-c6ff-4025-9193-9f09f4488bc9" containerName="revoke-keys"
Oct 30 22:10:16.216167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.215912    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c671a13e-3166-4293-a989-956f2f6f2873" containerName="istio-proxy"
Oct 30 22:10:16.221132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.221108    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.289753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.289714    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"]
Oct 30 22:10:16.327681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.327639    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"]
Oct 30 22:10:16.332678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.332646    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4"]
Oct 30 22:10:16.340928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.340896    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cp87\" (UniqueName: \"kubernetes.io/projected/56020270-adad-4456-951a-1d8926a44e11-kube-api-access-7cp87\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.341064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.340975    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56020270-adad-4456-951a-1d8926a44e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.341116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.341094    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56020270-adad-4456-951a-1d8926a44e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.445835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.445729    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cp87\" (UniqueName: \"kubernetes.io/projected/56020270-adad-4456-951a-1d8926a44e11-kube-api-access-7cp87\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.446013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.445844    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56020270-adad-4456-951a-1d8926a44e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.446013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.445917    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56020270-adad-4456-951a-1d8926a44e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.446769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.446725    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56020270-adad-4456-951a-1d8926a44e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.449173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.449148    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56020270-adad-4456-951a-1d8926a44e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.509944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.509906    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cp87\" (UniqueName: \"kubernetes.io/projected/56020270-adad-4456-951a-1d8926a44e11-kube-api-access-7cp87\") pod \"maas-api-revoke-sub-d88c3c4d-s475l\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.537224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.537174    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:16.567802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.567753    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"]
Oct 30 22:10:16.574372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.574335    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr"]
Oct 30 22:10:16.648259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.648176    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47fd3610-6642-485b-9c49-4eb2b2912eee" path="/var/lib/kubelet/pods/47fd3610-6642-485b-9c49-4eb2b2912eee/volumes"
Oct 30 22:10:16.648851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.648825    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72911b4-60ba-41ea-a998-be27d168c586" path="/var/lib/kubelet/pods/a72911b4-60ba-41ea-a998-be27d168c586/volumes"
Oct 30 22:10:16.649513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.649491    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae72439-1274-4abd-8182-6f918935d93f" path="/var/lib/kubelet/pods/cae72439-1274-4abd-8182-6f918935d93f/volumes"
Oct 30 22:10:16.650650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.650629    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a7728b-68ff-4812-b8ea-e7ea9385e08c" path="/var/lib/kubelet/pods/e5a7728b-68ff-4812-b8ea-e7ea9385e08c/volumes"
Oct 30 22:10:16.697415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.697308    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"]
Oct 30 22:10:16.702452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.702417    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp"]
Oct 30 22:10:16.931478 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:16.931432    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56020270_adad_4456_951a_1d8926a44e11.slice/crio-dd43f00fa5940a97b90b23e84cae92fb18ef798983dd5177878e53b721a14735 WatchSource:0}: Error finding container dd43f00fa5940a97b90b23e84cae92fb18ef798983dd5177878e53b721a14735: Status 404 returned error can't find the container with id dd43f00fa5940a97b90b23e84cae92fb18ef798983dd5177878e53b721a14735
Oct 30 22:10:16.933126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:16.932887    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"]
Oct 30 22:10:17.080604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.080562    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:17.262064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.262027    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-xks-ca\") pod \"ea3a6021-8bbf-480a-ace7-98426d786963\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") "
Oct 30 22:10:17.262300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.262268    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3a6021-8bbf-480a-ace7-98426d786963-kube-api-access-vfq2f\" (UniqueName: \"kubernetes.io/projected/ea3a6021-8bbf-480a-ace7-98426d786963-kube-api-access-vfq2f\") pod \"ea3a6021-8bbf-480a-ace7-98426d786963\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") "
Oct 30 22:10:17.262608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.262569    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-openshift-ca\") pod \"ea3a6021-8bbf-480a-ace7-98426d786963\" (UID: \"ea3a6021-8bbf-480a-ace7-98426d786963\") "
Oct 30 22:10:17.262958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.262937    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-openshift-ca" pod "ea3a6021-8bbf-480a-ace7-98426d786963" (UID: "ea3a6021-8bbf-480a-ace7-98426d786963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.263267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.263226    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.265103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.265079    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-xks-ca" pod "ea3a6021-8bbf-480a-ace7-98426d786963" (UID: "ea3a6021-8bbf-480a-ace7-98426d786963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:17.265411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.265387    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3a6021-8bbf-480a-ace7-98426d786963-kube-api-access-vfq2f" pod "ea3a6021-8bbf-480a-ace7-98426d786963" (UID: "ea3a6021-8bbf-480a-ace7-98426d786963"). InnerVolumeSpecName "kube-api-access-vfq2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.364781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.364740    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3a6021-8bbf-480a-ace7-98426d786963-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.364781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.364772    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfq2f\" (UniqueName: \"kubernetes.io/projected/ea3a6021-8bbf-480a-ace7-98426d786963-kube-api-access-vfq2f\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.887652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.887611    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l" event={"ID":"56020270-adad-4456-951a-1d8926a44e11","Type":"ContainerStarted","Data":"e50fc8b9578544916549927cdce26b1ec68e91dd0c779b1434308250faca5b28"}
Oct 30 22:10:17.887882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.887674    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l" event={"ID":"56020270-adad-4456-951a-1d8926a44e11","Type":"ContainerStarted","Data":"dd43f00fa5940a97b90b23e84cae92fb18ef798983dd5177878e53b721a14735"}
Oct 30 22:10:17.889524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.889501    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"
Oct 30 22:10:17.889656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.889542    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n" event={"ID":"ea3a6021-8bbf-480a-ace7-98426d786963","Type":"ContainerDied","Data":"f7e290e07a77ae0f5a06d50e3668ffb373e8df27faca8d36a378400c3a2bb12f"}
Oct 30 22:10:17.889656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:17.889585    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7e290e07a77ae0f5a06d50e3668ffb373e8df27faca8d36a378400c3a2bb12f"
Oct 30 22:10:18.023944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:18.023899    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l" podStartSLOduration=2.02388374 podStartE2EDuration="2.02388374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:18.022015144 +0000 UTC m=+1229.546157908" watchObservedRunningTime="2026-10-30 22:10:18.02388374 +0000 UTC m=+1229.548026504"
Oct 30 22:10:18.650034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:18.649999    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c671a13e-3166-4293-a989-956f2f6f2873" path="/var/lib/kubelet/pods/c671a13e-3166-4293-a989-956f2f6f2873/volumes"
Oct 30 22:10:18.918706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:18.918610    5031 generic.go:356] "Generic (PLEG): container finished" podID="56020270-adad-4456-951a-1d8926a44e11" containerID="e50fc8b9578544916549927cdce26b1ec68e91dd0c779b1434308250faca5b28" exitCode=0
Oct 30 22:10:18.918901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:18.918702    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l" event={"ID":"56020270-adad-4456-951a-1d8926a44e11","Type":"ContainerDied","Data":"e50fc8b9578544916549927cdce26b1ec68e91dd0c779b1434308250faca5b28"}
Oct 30 22:10:20.101337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.101310    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:20.108849 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.108817    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56020270-adad-4456-951a-1d8926a44e11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56020270-adad-4456-951a-1d8926a44e11-maas-api-openshift-ca\") pod \"56020270-adad-4456-951a-1d8926a44e11\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") "
Oct 30 22:10:20.109041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.108882    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56020270-adad-4456-951a-1d8926a44e11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56020270-adad-4456-951a-1d8926a44e11-maas-api-xks-ca\") pod \"56020270-adad-4456-951a-1d8926a44e11\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") "
Oct 30 22:10:20.109123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.109104    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56020270-adad-4456-951a-1d8926a44e11-kube-api-access-7cp87\" (UniqueName: \"kubernetes.io/projected/56020270-adad-4456-951a-1d8926a44e11-kube-api-access-7cp87\") pod \"56020270-adad-4456-951a-1d8926a44e11\" (UID: \"56020270-adad-4456-951a-1d8926a44e11\") "
Oct 30 22:10:20.109331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.109194    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56020270-adad-4456-951a-1d8926a44e11-maas-api-openshift-ca" pod "56020270-adad-4456-951a-1d8926a44e11" (UID: "56020270-adad-4456-951a-1d8926a44e11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:20.109834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.109809    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56020270-adad-4456-951a-1d8926a44e11-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.111481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.111457    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56020270-adad-4456-951a-1d8926a44e11-maas-api-xks-ca" pod "56020270-adad-4456-951a-1d8926a44e11" (UID: "56020270-adad-4456-951a-1d8926a44e11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:20.111911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.111890    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56020270-adad-4456-951a-1d8926a44e11-kube-api-access-7cp87" pod "56020270-adad-4456-951a-1d8926a44e11" (UID: "56020270-adad-4456-951a-1d8926a44e11"). InnerVolumeSpecName "kube-api-access-7cp87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:20.211719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.211624    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cp87\" (UniqueName: \"kubernetes.io/projected/56020270-adad-4456-951a-1d8926a44e11-kube-api-access-7cp87\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.211719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.211668    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56020270-adad-4456-951a-1d8926a44e11-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.545537 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:10:20.545440    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:10:20.652540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.652073    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:20.859561 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.859520    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"]
Oct 30 22:10:20.860297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.860282    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ea3a6021-8bbf-480a-ace7-98426d786963" containerName="revoke-keys"
Oct 30 22:10:20.860355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.860298    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3a6021-8bbf-480a-ace7-98426d786963" containerName="revoke-keys"
Oct 30 22:10:20.860355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.860347    5031 cpu_manager.go:414] "containerMap: removing container" podUID="56020270-adad-4456-951a-1d8926a44e11" containerName="revoke-keys"
Oct 30 22:10:20.860355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.860352    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56020270-adad-4456-951a-1d8926a44e11" containerName="revoke-keys"
Oct 30 22:10:20.860458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.860449    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3a6021-8bbf-480a-ace7-98426d786963" containerName="revoke-keys"
Oct 30 22:10:20.860494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.860468    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="56020270-adad-4456-951a-1d8926a44e11" containerName="revoke-keys"
Oct 30 22:10:20.864460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.864445    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:20.923935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.923901    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"]
Oct 30 22:10:20.924460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.924432    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:20.924700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.924679    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gg5\" (UniqueName: \"kubernetes.io/projected/5155d09e-f1c7-4236-82ad-51142f45a870-kube-api-access-54gg5\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:20.924856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.924838    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:20.946383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.946337    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l" event={"ID":"56020270-adad-4456-951a-1d8926a44e11","Type":"ContainerDied","Data":"dd43f00fa5940a97b90b23e84cae92fb18ef798983dd5177878e53b721a14735"}
Oct 30 22:10:20.946590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.946407    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd43f00fa5940a97b90b23e84cae92fb18ef798983dd5177878e53b721a14735"
Oct 30 22:10:20.946590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:20.946483    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"
Oct 30 22:10:21.026922 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.026873    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gg5\" (UniqueName: \"kubernetes.io/projected/5155d09e-f1c7-4236-82ad-51142f45a870-kube-api-access-54gg5\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:21.027142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.027020    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:21.027255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.027157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:21.027808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.027775    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:21.029985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.029956    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:21.065963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.065924    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"]
Oct 30 22:10:21.071053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.070959    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8788fd31-qhccs"]
Oct 30 22:10:21.094372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.094338    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gg5\" (UniqueName: \"kubernetes.io/projected/5155d09e-f1c7-4236-82ad-51142f45a870-kube-api-access-54gg5\") pod \"maas-api-revoke-sub-3583a9c1-gs6tn\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:21.180563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.180478    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:21.583743 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:21.583705    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5155d09e_f1c7_4236_82ad_51142f45a870.slice/crio-298c0dbcecd40557f97ce6b0fb059d8043edc671cd5dca49911372fc04f24368 WatchSource:0}: Error finding container 298c0dbcecd40557f97ce6b0fb059d8043edc671cd5dca49911372fc04f24368: Status 404 returned error can't find the container with id 298c0dbcecd40557f97ce6b0fb059d8043edc671cd5dca49911372fc04f24368
Oct 30 22:10:21.584763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.584735    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"]
Oct 30 22:10:21.958472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:21.958434    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn" event={"ID":"5155d09e-f1c7-4236-82ad-51142f45a870","Type":"ContainerStarted","Data":"298c0dbcecd40557f97ce6b0fb059d8043edc671cd5dca49911372fc04f24368"}
Oct 30 22:10:22.084528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.084491    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"]
Oct 30 22:10:22.092385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.090985    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.155002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.154966    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"]
Oct 30 22:10:22.157029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.155328    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zjw\" (UniqueName: \"kubernetes.io/projected/cd123a07-c372-4eba-9616-e6598c3a47cc-kube-api-access-89zjw\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.157029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.155391    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.157029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.155658    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.257654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.257549    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zjw\" (UniqueName: \"kubernetes.io/projected/cd123a07-c372-4eba-9616-e6598c3a47cc-kube-api-access-89zjw\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.257654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.257607    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.258188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.257880    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.258318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.258297    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.260466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.260436    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.321818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.321772    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zjw\" (UniqueName: \"kubernetes.io/projected/cd123a07-c372-4eba-9616-e6598c3a47cc-kube-api-access-89zjw\") pod \"maas-api-revoke-sub-7daae72a-dj7vr\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.423663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.423622    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:22.426625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.426593    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"]
Oct 30 22:10:22.432973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.432942    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.494765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.493343    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"]
Oct 30 22:10:22.564542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.564449    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.564710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.564577    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ww4t\" (UniqueName: \"kubernetes.io/projected/e803b626-79ec-4d21-ad84-677b3f989072-kube-api-access-5ww4t\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.564767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.564726    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.652247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.652177    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b69f820-692e-40ed-b295-3b7655bad054" path="/var/lib/kubelet/pods/8b69f820-692e-40ed-b295-3b7655bad054/volumes"
Oct 30 22:10:22.666452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.666418    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ww4t\" (UniqueName: \"kubernetes.io/projected/e803b626-79ec-4d21-ad84-677b3f989072-kube-api-access-5ww4t\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.666645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.666540    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.666797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.666781    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.667252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.667195    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.669435 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.669413    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.730787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.730753    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ww4t\" (UniqueName: \"kubernetes.io/projected/e803b626-79ec-4d21-ad84-677b3f989072-kube-api-access-5ww4t\") pod \"maas-api-revoke-sub-972f24ad-qqld7\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.779955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.779921    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:22.827229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.827163    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"]
Oct 30 22:10:22.972394 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.972353    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr" event={"ID":"cd123a07-c372-4eba-9616-e6598c3a47cc","Type":"ContainerStarted","Data":"3ffacde1177b4007b166ca20542d27f71972f9e985540d6b193065aa1d2e29d2"}
Oct 30 22:10:22.974315 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.974287    5031 generic.go:356] "Generic (PLEG): container finished" podID="5155d09e-f1c7-4236-82ad-51142f45a870" containerID="791741d047ffbd97d6f9ccbbd8e2737c1221ef8f187c481bc3ea8cf25e529565" exitCode=0
Oct 30 22:10:22.974444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:22.974369    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn" event={"ID":"5155d09e-f1c7-4236-82ad-51142f45a870","Type":"ContainerDied","Data":"791741d047ffbd97d6f9ccbbd8e2737c1221ef8f187c481bc3ea8cf25e529565"}
Oct 30 22:10:23.182163 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:23.182129    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode803b626_79ec_4d21_ad84_677b3f989072.slice/crio-ec339135da4f6f4fc71f341f83bede8b1ccc4778db4a011249ce83da372eec52 WatchSource:0}: Error finding container ec339135da4f6f4fc71f341f83bede8b1ccc4778db4a011249ce83da372eec52: Status 404 returned error can't find the container with id ec339135da4f6f4fc71f341f83bede8b1ccc4778db4a011249ce83da372eec52
Oct 30 22:10:23.183007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:23.182978    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"]
Oct 30 22:10:23.986757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:23.986725    5031 generic.go:356] "Generic (PLEG): container finished" podID="cd123a07-c372-4eba-9616-e6598c3a47cc" containerID="f5772fd9b069a5a0df153538217ee2d8647e1315db35a3e53c3a9236eb7a2aca" exitCode=0
Oct 30 22:10:23.987223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:23.986867    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr" event={"ID":"cd123a07-c372-4eba-9616-e6598c3a47cc","Type":"ContainerDied","Data":"f5772fd9b069a5a0df153538217ee2d8647e1315db35a3e53c3a9236eb7a2aca"}
Oct 30 22:10:23.988835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:23.988811    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7" event={"ID":"e803b626-79ec-4d21-ad84-677b3f989072","Type":"ContainerStarted","Data":"51138bb1a84d83d4ce1acea120aef47fe11fd5549acd42e39ce01e1b9763fc4c"}
Oct 30 22:10:23.988953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:23.988855    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7" event={"ID":"e803b626-79ec-4d21-ad84-677b3f989072","Type":"ContainerStarted","Data":"ec339135da4f6f4fc71f341f83bede8b1ccc4778db4a011249ce83da372eec52"}
Oct 30 22:10:24.197280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.197248    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:24.235197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.235070    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7" podStartSLOduration=2.235055248 podStartE2EDuration="2.235055248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:24.233662908 +0000 UTC m=+1235.757805697" watchObservedRunningTime="2026-10-30 22:10:24.235055248 +0000 UTC m=+1235.759198011"
Oct 30 22:10:24.291835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.291808    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5155d09e-f1c7-4236-82ad-51142f45a870-kube-api-access-54gg5\" (UniqueName: \"kubernetes.io/projected/5155d09e-f1c7-4236-82ad-51142f45a870-kube-api-access-54gg5\") pod \"5155d09e-f1c7-4236-82ad-51142f45a870\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") "
Oct 30 22:10:24.291989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.291975    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-xks-ca\") pod \"5155d09e-f1c7-4236-82ad-51142f45a870\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") "
Oct 30 22:10:24.292047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.292035    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-openshift-ca\") pod \"5155d09e-f1c7-4236-82ad-51142f45a870\" (UID: \"5155d09e-f1c7-4236-82ad-51142f45a870\") "
Oct 30 22:10:24.292481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.292455    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-openshift-ca" pod "5155d09e-f1c7-4236-82ad-51142f45a870" (UID: "5155d09e-f1c7-4236-82ad-51142f45a870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:24.292753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.292695    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:24.294190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.294157    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5155d09e-f1c7-4236-82ad-51142f45a870-kube-api-access-54gg5" pod "5155d09e-f1c7-4236-82ad-51142f45a870" (UID: "5155d09e-f1c7-4236-82ad-51142f45a870"). InnerVolumeSpecName "kube-api-access-54gg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:24.294414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.294392    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-xks-ca" pod "5155d09e-f1c7-4236-82ad-51142f45a870" (UID: "5155d09e-f1c7-4236-82ad-51142f45a870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:24.394631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.394586    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54gg5\" (UniqueName: \"kubernetes.io/projected/5155d09e-f1c7-4236-82ad-51142f45a870-kube-api-access-54gg5\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:24.394631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.394633    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155d09e-f1c7-4236-82ad-51142f45a870-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:24.398628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.398591    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"]
Oct 30 22:10:24.399814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.399770    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5155d09e-f1c7-4236-82ad-51142f45a870" containerName="revoke-keys"
Oct 30 22:10:24.399814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.399806    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5155d09e-f1c7-4236-82ad-51142f45a870" containerName="revoke-keys"
Oct 30 22:10:24.400090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.400072    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5155d09e-f1c7-4236-82ad-51142f45a870" containerName="revoke-keys"
Oct 30 22:10:24.430274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.404948    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.477510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.477474    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"]
Oct 30 22:10:24.496147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.496053    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.496368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.496298    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.496428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.496388    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/81f46e03-5714-4437-9a87-9aac2d445a80-kube-api-access-6rqbk\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.598546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.598509    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.598780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.598633    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.598780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.598681    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/81f46e03-5714-4437-9a87-9aac2d445a80-kube-api-access-6rqbk\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.599374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.599343    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.601342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.601320    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.663611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.663578    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/81f46e03-5714-4437-9a87-9aac2d445a80-kube-api-access-6rqbk\") pod \"maas-api-revoke-sub-3f8481e9-8pxs9\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.748427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.748334    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:24.998944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.998861    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"
Oct 30 22:10:24.998944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.998882    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn" event={"ID":"5155d09e-f1c7-4236-82ad-51142f45a870","Type":"ContainerDied","Data":"298c0dbcecd40557f97ce6b0fb059d8043edc671cd5dca49911372fc04f24368"}
Oct 30 22:10:24.999454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:24.998973    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298c0dbcecd40557f97ce6b0fb059d8043edc671cd5dca49911372fc04f24368"
Oct 30 22:10:25.000804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.000777    5031 generic.go:356] "Generic (PLEG): container finished" podID="e803b626-79ec-4d21-ad84-677b3f989072" containerID="51138bb1a84d83d4ce1acea120aef47fe11fd5549acd42e39ce01e1b9763fc4c" exitCode=0
Oct 30 22:10:25.001013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.000978    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7" event={"ID":"e803b626-79ec-4d21-ad84-677b3f989072","Type":"ContainerDied","Data":"51138bb1a84d83d4ce1acea120aef47fe11fd5549acd42e39ce01e1b9763fc4c"}
Oct 30 22:10:25.063544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.063507    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"]
Oct 30 22:10:25.117278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.115332    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03613f5b-4stxl"]
Oct 30 22:10:25.185323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.185295    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:25.186153 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:25.186123    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f46e03_5714_4437_9a87_9aac2d445a80.slice/crio-30774d8614ea3409fc5321d4b7b4a987c7674b4b259be2993d3054bc87072c31 WatchSource:0}: Error finding container 30774d8614ea3409fc5321d4b7b4a987c7674b4b259be2993d3054bc87072c31: Status 404 returned error can't find the container with id 30774d8614ea3409fc5321d4b7b4a987c7674b4b259be2993d3054bc87072c31
Oct 30 22:10:25.187031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.187003    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"]
Oct 30 22:10:25.311375 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.311340    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd123a07-c372-4eba-9616-e6598c3a47cc-kube-api-access-89zjw\" (UniqueName: \"kubernetes.io/projected/cd123a07-c372-4eba-9616-e6598c3a47cc-kube-api-access-89zjw\") pod \"cd123a07-c372-4eba-9616-e6598c3a47cc\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") "
Oct 30 22:10:25.311578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.311453    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-openshift-ca\") pod \"cd123a07-c372-4eba-9616-e6598c3a47cc\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") "
Oct 30 22:10:25.311636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.311593    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-xks-ca\") pod \"cd123a07-c372-4eba-9616-e6598c3a47cc\" (UID: \"cd123a07-c372-4eba-9616-e6598c3a47cc\") "
Oct 30 22:10:25.311933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.311903    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-openshift-ca" pod "cd123a07-c372-4eba-9616-e6598c3a47cc" (UID: "cd123a07-c372-4eba-9616-e6598c3a47cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:25.312647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.312623    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:25.313927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.313891    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd123a07-c372-4eba-9616-e6598c3a47cc-kube-api-access-89zjw" pod "cd123a07-c372-4eba-9616-e6598c3a47cc" (UID: "cd123a07-c372-4eba-9616-e6598c3a47cc"). InnerVolumeSpecName "kube-api-access-89zjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:25.314043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.314001    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-xks-ca" pod "cd123a07-c372-4eba-9616-e6598c3a47cc" (UID: "cd123a07-c372-4eba-9616-e6598c3a47cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:25.341152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.341113    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"]
Oct 30 22:10:25.342049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.342024    5031 cpu_manager.go:414] "containerMap: removing container" podUID="cd123a07-c372-4eba-9616-e6598c3a47cc" containerName="revoke-keys"
Oct 30 22:10:25.342049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.342048    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd123a07-c372-4eba-9616-e6598c3a47cc" containerName="revoke-keys"
Oct 30 22:10:25.342289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.342241    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd123a07-c372-4eba-9616-e6598c3a47cc" containerName="revoke-keys"
Oct 30 22:10:25.348027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.347985    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.404137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.404055    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"]
Oct 30 22:10:25.428516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.421929    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79j6f\" (UniqueName: \"kubernetes.io/projected/87e5101e-c304-4dcd-901c-030ab139c485-kube-api-access-79j6f\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.428516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.426739    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.428516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.427561    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.428516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.427697    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89zjw\" (UniqueName: \"kubernetes.io/projected/cd123a07-c372-4eba-9616-e6598c3a47cc-kube-api-access-89zjw\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:25.428516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.427715    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd123a07-c372-4eba-9616-e6598c3a47cc-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:25.529599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.529557    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.529829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.529735    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.529829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.529781    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79j6f\" (UniqueName: \"kubernetes.io/projected/87e5101e-c304-4dcd-901c-030ab139c485-kube-api-access-79j6f\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.530340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.530317    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.532431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.532404    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.606163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.606132    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79j6f\" (UniqueName: \"kubernetes.io/projected/87e5101e-c304-4dcd-901c-030ab139c485-kube-api-access-79j6f\") pod \"maas-api-revoke-sub-30f6ea7a-2q642\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.668402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.668371    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:25.824767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:25.824723    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:26.019898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.019795    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9" event={"ID":"81f46e03-5714-4437-9a87-9aac2d445a80","Type":"ContainerStarted","Data":"ea3cdf2d17ea8352ce04e2c86fd45fce91cbb4db7c00a4bc34217f69cd80fbd7"}
Oct 30 22:10:26.020416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.019939    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9" event={"ID":"81f46e03-5714-4437-9a87-9aac2d445a80","Type":"ContainerStarted","Data":"30774d8614ea3409fc5321d4b7b4a987c7674b4b259be2993d3054bc87072c31"}
Oct 30 22:10:26.022747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.022716    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr" event={"ID":"cd123a07-c372-4eba-9616-e6598c3a47cc","Type":"ContainerDied","Data":"3ffacde1177b4007b166ca20542d27f71972f9e985540d6b193065aa1d2e29d2"}
Oct 30 22:10:26.022747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.022746    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"
Oct 30 22:10:26.022953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.022770    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ffacde1177b4007b166ca20542d27f71972f9e985540d6b193065aa1d2e29d2"
Oct 30 22:10:26.062808 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:26.062765    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e5101e_c304_4dcd_901c_030ab139c485.slice/crio-46703ff544e16a59fd758e7a49c1edb1fcc5ea06091c5efeb8a0bb350b9e9854 WatchSource:0}: Error finding container 46703ff544e16a59fd758e7a49c1edb1fcc5ea06091c5efeb8a0bb350b9e9854: Status 404 returned error can't find the container with id 46703ff544e16a59fd758e7a49c1edb1fcc5ea06091c5efeb8a0bb350b9e9854
Oct 30 22:10:26.064265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.064241    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"]
Oct 30 22:10:26.152274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.151652    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9" podStartSLOduration=2.151632236 podStartE2EDuration="2.151632236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:26.150963484 +0000 UTC m=+1237.675106250" watchObservedRunningTime="2026-10-30 22:10:26.151632236 +0000 UTC m=+1237.675775000"
Oct 30 22:10:26.209370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.209341    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:10:26.219728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.219692    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:26.346924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.346261    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e803b626-79ec-4d21-ad84-677b3f989072-kube-api-access-5ww4t\" (UniqueName: \"kubernetes.io/projected/e803b626-79ec-4d21-ad84-677b3f989072-kube-api-access-5ww4t\") pod \"e803b626-79ec-4d21-ad84-677b3f989072\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") "
Oct 30 22:10:26.347156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.347023    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-xks-ca\") pod \"e803b626-79ec-4d21-ad84-677b3f989072\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") "
Oct 30 22:10:26.347156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.347121    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-openshift-ca\") pod \"e803b626-79ec-4d21-ad84-677b3f989072\" (UID: \"e803b626-79ec-4d21-ad84-677b3f989072\") "
Oct 30 22:10:26.347659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.347626    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-openshift-ca" pod "e803b626-79ec-4d21-ad84-677b3f989072" (UID: "e803b626-79ec-4d21-ad84-677b3f989072"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:26.348128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.348106    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:26.349281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.349255    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e803b626-79ec-4d21-ad84-677b3f989072-kube-api-access-5ww4t" pod "e803b626-79ec-4d21-ad84-677b3f989072" (UID: "e803b626-79ec-4d21-ad84-677b3f989072"). InnerVolumeSpecName "kube-api-access-5ww4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:26.349609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.349585    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-xks-ca" pod "e803b626-79ec-4d21-ad84-677b3f989072" (UID: "e803b626-79ec-4d21-ad84-677b3f989072"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:26.449406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.449371    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e803b626-79ec-4d21-ad84-677b3f989072-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:26.449406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.449403    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ww4t\" (UniqueName: \"kubernetes.io/projected/e803b626-79ec-4d21-ad84-677b3f989072-kube-api-access-5ww4t\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:26.648634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:26.648599    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b048e45-b26f-4a64-a26c-4d6eea108f2f" path="/var/lib/kubelet/pods/7b048e45-b26f-4a64-a26c-4d6eea108f2f/volumes"
Oct 30 22:10:27.105865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.105821    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7" event={"ID":"e803b626-79ec-4d21-ad84-677b3f989072","Type":"ContainerDied","Data":"ec339135da4f6f4fc71f341f83bede8b1ccc4778db4a011249ce83da372eec52"}
Oct 30 22:10:27.106645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.106565    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"
Oct 30 22:10:27.115069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.115031    5031 generic.go:356] "Generic (PLEG): container finished" podID="81f46e03-5714-4437-9a87-9aac2d445a80" containerID="ea3cdf2d17ea8352ce04e2c86fd45fce91cbb4db7c00a4bc34217f69cd80fbd7" exitCode=0
Oct 30 22:10:27.115621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.109910    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec339135da4f6f4fc71f341f83bede8b1ccc4778db4a011249ce83da372eec52"
Oct 30 22:10:27.115828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.115807    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642" event={"ID":"87e5101e-c304-4dcd-901c-030ab139c485","Type":"ContainerStarted","Data":"9617f8c05c7d4e5a6ff407ad86f62eafd983d2d5abbd449277e75af5c4d46669"}
Oct 30 22:10:27.115997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.115979    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642" event={"ID":"87e5101e-c304-4dcd-901c-030ab139c485","Type":"ContainerStarted","Data":"46703ff544e16a59fd758e7a49c1edb1fcc5ea06091c5efeb8a0bb350b9e9854"}
Oct 30 22:10:27.116169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.116152    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9" event={"ID":"81f46e03-5714-4437-9a87-9aac2d445a80","Type":"ContainerDied","Data":"ea3cdf2d17ea8352ce04e2c86fd45fce91cbb4db7c00a4bc34217f69cd80fbd7"}
Oct 30 22:10:27.310231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:27.283687    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642" podStartSLOduration=2.283666903 podStartE2EDuration="2.283666903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:27.248689289 +0000 UTC m=+1238.772832054" watchObservedRunningTime="2026-10-30 22:10:27.283666903 +0000 UTC m=+1238.807809667"
Oct 30 22:10:28.200831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:28.200795    5031 generic.go:356] "Generic (PLEG): container finished" podID="87e5101e-c304-4dcd-901c-030ab139c485" containerID="9617f8c05c7d4e5a6ff407ad86f62eafd983d2d5abbd449277e75af5c4d46669" exitCode=0
Oct 30 22:10:28.202165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:28.202130    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642" event={"ID":"87e5101e-c304-4dcd-901c-030ab139c485","Type":"ContainerDied","Data":"9617f8c05c7d4e5a6ff407ad86f62eafd983d2d5abbd449277e75af5c4d46669"}
Oct 30 22:10:28.430249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:28.414858    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.101238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.100331    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.124237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.122322    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.148236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.143377    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.148236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.143935    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.156284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.153665    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.243233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.231649    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.532331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.527365    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:29.532331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.527831    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:10:29.532331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.528239    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:10:29.889609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:29.888626    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.240298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.210420    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.407677 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.212381    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.409287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.213195    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.435763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.213248    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.436251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.213307    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.436570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.268558    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.436862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.411447    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:10:30.462948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.462919    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:10:30.463226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.463190    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:10:30.471621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.463391    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:10:30.500768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.500674    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:10:30.513236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.501009    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:10:30.563244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.562090    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:10:30.579238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.562034    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.614667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.614625    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:10:30.773541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.773448    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:30.927345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:30.911339    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.478244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.477367    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f46e03-5714-4437-9a87-9aac2d445a80-kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/81f46e03-5714-4437-9a87-9aac2d445a80-kube-api-access-6rqbk\") pod \"81f46e03-5714-4437-9a87-9aac2d445a80\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") "
Oct 30 22:10:31.561681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.561630    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.587237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.587065    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"
Oct 30 22:10:31.587675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.587634    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.596245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.590576    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.832236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.832084    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:10:31.971240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.677426    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:10:32.886309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.892329    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:10:32.897483 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:10:31.897948    5031 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.253s"
Oct 30 22:10:32.992255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.992132    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9" event={"ID":"81f46e03-5714-4437-9a87-9aac2d445a80","Type":"ContainerDied","Data":"30774d8614ea3409fc5321d4b7b4a987c7674b4b259be2993d3054bc87072c31"}
Oct 30 22:10:32.992543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.992522    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30774d8614ea3409fc5321d4b7b4a987c7674b4b259be2993d3054bc87072c31"
Oct 30 22:10:32.992823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.992811    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:10:33.031850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.013632    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:10:33.095415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.702158    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:33.095683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.921475    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:10:33.095821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.974610    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:10:33.095911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.975024    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:10:33.096001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.983019    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-openshift-ca\") pod \"81f46e03-5714-4437-9a87-9aac2d445a80\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") "
Oct 30 22:10:33.096142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.096126    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-xks-ca\") pod \"81f46e03-5714-4437-9a87-9aac2d445a80\" (UID: \"81f46e03-5714-4437-9a87-9aac2d445a80\") "
Oct 30 22:10:33.096457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.637553    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:10:33.096590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:32.377479    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f46e03-5714-4437-9a87-9aac2d445a80-kube-api-access-6rqbk" pod "81f46e03-5714-4437-9a87-9aac2d445a80" (UID: "81f46e03-5714-4437-9a87-9aac2d445a80"). InnerVolumeSpecName "kube-api-access-6rqbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:33.096678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:31.972248    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:33.101383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.101344    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-openshift-ca" pod "81f46e03-5714-4437-9a87-9aac2d445a80" (UID: "81f46e03-5714-4437-9a87-9aac2d445a80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:33.340351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.340312    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.340600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.340581    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqbk\" (UniqueName: \"kubernetes.io/projected/81f46e03-5714-4437-9a87-9aac2d445a80-kube-api-access-6rqbk\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.384321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.382186    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-xks-ca" pod "81f46e03-5714-4437-9a87-9aac2d445a80" (UID: "81f46e03-5714-4437-9a87-9aac2d445a80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:33.402643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.393766    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.407158    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.408538    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.408583    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.408615    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.408744    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"]
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.408766    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642" event={"ID":"87e5101e-c304-4dcd-901c-030ab139c485","Type":"ContainerDied","Data":"46703ff544e16a59fd758e7a49c1edb1fcc5ea06091c5efeb8a0bb350b9e9854"}
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.408795    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46703ff544e16a59fd758e7a49c1edb1fcc5ea06091c5efeb8a0bb350b9e9854"
Oct 30 22:10:33.410230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.408827    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:10:33.444317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.435097    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:10:33.447425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.447396    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5c6f4-84hfb"]
Oct 30 22:10:33.447652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.447639    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:10:33.449681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.449655    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f46e03-5714-4437-9a87-9aac2d445a80-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.466351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.466313    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"]
Oct 30 22:10:33.469656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.467624    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:10:33.469656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.467684    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:10:33.469656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.468505    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:10:33.474469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.474417    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d356c69e-nh86l"]
Oct 30 22:10:33.493261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.493231    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"]
Oct 30 22:10:33.494887 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:10:33.494444    5031 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f46e03_5714_4437_9a87_9aac2d445a80.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:10:33.498560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.498532    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e60243-dxmqf"]
Oct 30 22:10:33.552193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.552153    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e5101e-c304-4dcd-901c-030ab139c485-kube-api-access-79j6f\" (UniqueName: \"kubernetes.io/projected/87e5101e-c304-4dcd-901c-030ab139c485-kube-api-access-79j6f\") pod \"87e5101e-c304-4dcd-901c-030ab139c485\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") "
Oct 30 22:10:33.552615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.552329    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-openshift-ca\") pod \"87e5101e-c304-4dcd-901c-030ab139c485\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") "
Oct 30 22:10:33.552615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.552364    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-xks-ca\") pod \"87e5101e-c304-4dcd-901c-030ab139c485\" (UID: \"87e5101e-c304-4dcd-901c-030ab139c485\") "
Oct 30 22:10:33.553113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.553078    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-openshift-ca" pod "87e5101e-c304-4dcd-901c-030ab139c485" (UID: "87e5101e-c304-4dcd-901c-030ab139c485"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:33.553662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.553643    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.556053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.556031    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-xks-ca" pod "87e5101e-c304-4dcd-901c-030ab139c485" (UID: "87e5101e-c304-4dcd-901c-030ab139c485"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:33.556914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.556889    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e5101e-c304-4dcd-901c-030ab139c485-kube-api-access-79j6f" pod "87e5101e-c304-4dcd-901c-030ab139c485" (UID: "87e5101e-c304-4dcd-901c-030ab139c485"). InnerVolumeSpecName "kube-api-access-79j6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:33.655628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.655584    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e5101e-c304-4dcd-901c-030ab139c485-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:33.655628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:33.655623    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79j6f\" (UniqueName: \"kubernetes.io/projected/87e5101e-c304-4dcd-901c-030ab139c485-kube-api-access-79j6f\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:34.407111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:34.407080    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"
Oct 30 22:10:34.649467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:34.649428    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d697ea1-6c47-4633-b963-07b81688b100" path="/var/lib/kubelet/pods/0d697ea1-6c47-4633-b963-07b81688b100/volumes"
Oct 30 22:10:34.649986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:34.649969    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8" path="/var/lib/kubelet/pods/8947ad21-f13a-4ecc-bdae-9bf35fe4a2c8/volumes"
Oct 30 22:10:34.650416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:34.650401    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d4178c-eafe-4823-9772-85691ae354e7" path="/var/lib/kubelet/pods/a3d4178c-eafe-4823-9772-85691ae354e7/volumes"
Oct 30 22:10:37.233555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.233511    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"]
Oct 30 22:10:37.234658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234634    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e803b626-79ec-4d21-ad84-677b3f989072" containerName="revoke-keys"
Oct 30 22:10:37.234764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234660    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e803b626-79ec-4d21-ad84-677b3f989072" containerName="revoke-keys"
Oct 30 22:10:37.234764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234694    5031 cpu_manager.go:414] "containerMap: removing container" podUID="87e5101e-c304-4dcd-901c-030ab139c485" containerName="revoke-keys"
Oct 30 22:10:37.234764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234702    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e5101e-c304-4dcd-901c-030ab139c485" containerName="revoke-keys"
Oct 30 22:10:37.234764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234752    5031 cpu_manager.go:414] "containerMap: removing container" podUID="81f46e03-5714-4437-9a87-9aac2d445a80" containerName="revoke-keys"
Oct 30 22:10:37.234764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234760    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f46e03-5714-4437-9a87-9aac2d445a80" containerName="revoke-keys"
Oct 30 22:10:37.235039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234956    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f46e03-5714-4437-9a87-9aac2d445a80" containerName="revoke-keys"
Oct 30 22:10:37.235039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234974    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e803b626-79ec-4d21-ad84-677b3f989072" containerName="revoke-keys"
Oct 30 22:10:37.235039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.234989    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e5101e-c304-4dcd-901c-030ab139c485" containerName="revoke-keys"
Oct 30 22:10:37.240841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.240814    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.252351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.252318    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"]
Oct 30 22:10:37.259912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.259888    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.295051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.295022    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"]
Oct 30 22:10:37.295452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.295434    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:37.320569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.320533    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.320742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.320722    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzz4m\" (UniqueName: \"kubernetes.io/projected/68a6f9ac-660e-4883-a565-9c8d172f77ef-kube-api-access-rzz4m\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.320819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.320784    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.339482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.339448    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:10:37.339664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.339499    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:37.339664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.339555    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:37.345563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.345509    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"]
Oct 30 22:10:37.421971 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.421934    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzz4m\" (UniqueName: \"kubernetes.io/projected/68a6f9ac-660e-4883-a565-9c8d172f77ef-kube-api-access-rzz4m\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.422201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.421995    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.422201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.422053    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.422201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.422156    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzjv\" (UniqueName: \"kubernetes.io/projected/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-kube-api-access-5wzjv\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.422201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.422189    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.422536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.422327    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.423117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.423090    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.425723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.425678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.486157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.486076    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzz4m\" (UniqueName: \"kubernetes.io/projected/68a6f9ac-660e-4883-a565-9c8d172f77ef-kube-api-access-rzz4m\") pod \"maas-api-revoke-sub-3f25334f-twjlx\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.523878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.523832    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.524055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.523963    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzjv\" (UniqueName: \"kubernetes.io/projected/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-kube-api-access-5wzjv\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.524055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.523993    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.524715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.524690    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.526890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.526860    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.559985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.559953    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:37.588262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.588230    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzjv\" (UniqueName: \"kubernetes.io/projected/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-kube-api-access-5wzjv\") pod \"maas-api-revoke-sub-2b3707f7-snhpb\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.697799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.696091    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:37.879489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.879456    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:37.966927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:37.966816    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"]
Oct 30 22:10:38.272741 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:38.272708    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6fb2fd_bed1_4122_a6d7_6a1f3f0e1f56.slice/crio-5c27ab98e8abc956ccc57464166e46277fad7f9ef6d1c4d24066e56e389e08c8 WatchSource:0}: Error finding container 5c27ab98e8abc956ccc57464166e46277fad7f9ef6d1c4d24066e56e389e08c8: Status 404 returned error can't find the container with id 5c27ab98e8abc956ccc57464166e46277fad7f9ef6d1c4d24066e56e389e08c8
Oct 30 22:10:38.273419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.273393    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"]
Oct 30 22:10:38.363531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.363499    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"]
Oct 30 22:10:38.370377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.370351    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.438444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.438403    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"]
Oct 30 22:10:38.453417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.450404    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.453590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.453448    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p4hq\" (UniqueName: \"kubernetes.io/projected/fb7cb0d7-3921-45ef-8d9c-11409121c349-kube-api-access-4p4hq\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.453590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.453511    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.474284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.474243    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx" event={"ID":"68a6f9ac-660e-4883-a565-9c8d172f77ef","Type":"ContainerStarted","Data":"12ddd20c1074f2fb61fde0880c83aeedef8af4689685fd224bb718d083be03ef"}
Oct 30 22:10:38.475948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.475916    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb" event={"ID":"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56","Type":"ContainerStarted","Data":"5c27ab98e8abc956ccc57464166e46277fad7f9ef6d1c4d24066e56e389e08c8"}
Oct 30 22:10:38.555382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.555338    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.555581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.555396    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p4hq\" (UniqueName: \"kubernetes.io/projected/fb7cb0d7-3921-45ef-8d9c-11409121c349-kube-api-access-4p4hq\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.555581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.555477    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.556404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.556358    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.559126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.559089    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.654112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.654085    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p4hq\" (UniqueName: \"kubernetes.io/projected/fb7cb0d7-3921-45ef-8d9c-11409121c349-kube-api-access-4p4hq\") pod \"maas-api-revoke-sub-82a960fb-7tshc\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:38.688783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:38.688741    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:39.088344 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:39.088313    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7cb0d7_3921_45ef_8d9c_11409121c349.slice/crio-92c3a3b0ce1dcce75dd9c8e77e023ca9b7720ece1d231eb5d6c54e05ccf745d9 WatchSource:0}: Error finding container 92c3a3b0ce1dcce75dd9c8e77e023ca9b7720ece1d231eb5d6c54e05ccf745d9: Status 404 returned error can't find the container with id 92c3a3b0ce1dcce75dd9c8e77e023ca9b7720ece1d231eb5d6c54e05ccf745d9
Oct 30 22:10:39.090154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:39.090133    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"]
Oct 30 22:10:39.487648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:39.487613    5031 generic.go:356] "Generic (PLEG): container finished" podID="68a6f9ac-660e-4883-a565-9c8d172f77ef" containerID="a6ddf494b8e58444c31a028e0e0964d9cec2cd056296232c0aae8826d737fc87" exitCode=0
Oct 30 22:10:39.488141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:39.487708    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx" event={"ID":"68a6f9ac-660e-4883-a565-9c8d172f77ef","Type":"ContainerDied","Data":"a6ddf494b8e58444c31a028e0e0964d9cec2cd056296232c0aae8826d737fc87"}
Oct 30 22:10:39.490042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:39.490018    5031 generic.go:356] "Generic (PLEG): container finished" podID="ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" containerID="8573ff26f70514a3e9a937846a0c27042b47e2a2007c4d316f35afd6e33b0c1b" exitCode=0
Oct 30 22:10:39.490265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:39.490110    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb" event={"ID":"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56","Type":"ContainerDied","Data":"8573ff26f70514a3e9a937846a0c27042b47e2a2007c4d316f35afd6e33b0c1b"}
Oct 30 22:10:39.491824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:39.491799    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc" event={"ID":"fb7cb0d7-3921-45ef-8d9c-11409121c349","Type":"ContainerStarted","Data":"92c3a3b0ce1dcce75dd9c8e77e023ca9b7720ece1d231eb5d6c54e05ccf745d9"}
Oct 30 22:10:40.503158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.503124    5031 generic.go:356] "Generic (PLEG): container finished" podID="fb7cb0d7-3921-45ef-8d9c-11409121c349" containerID="613b78914ea787d0cc02f1901267558146ddf1e94b0faba8dbf78bfabcc4415f" exitCode=0
Oct 30 22:10:40.503652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.503245    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc" event={"ID":"fb7cb0d7-3921-45ef-8d9c-11409121c349","Type":"ContainerDied","Data":"613b78914ea787d0cc02f1901267558146ddf1e94b0faba8dbf78bfabcc4415f"}
Oct 30 22:10:40.719681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.719655    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:40.785501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.785423    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:40.891786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.891715    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-xks-ca\") pod \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") "
Oct 30 22:10:40.891786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.891793    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-xks-ca\") pod \"68a6f9ac-660e-4883-a565-9c8d172f77ef\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") "
Oct 30 22:10:40.892025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.891938    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-openshift-ca\") pod \"68a6f9ac-660e-4883-a565-9c8d172f77ef\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") "
Oct 30 22:10:40.892069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.892037    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a6f9ac-660e-4883-a565-9c8d172f77ef-kube-api-access-rzz4m\" (UniqueName: \"kubernetes.io/projected/68a6f9ac-660e-4883-a565-9c8d172f77ef-kube-api-access-rzz4m\") pod \"68a6f9ac-660e-4883-a565-9c8d172f77ef\" (UID: \"68a6f9ac-660e-4883-a565-9c8d172f77ef\") "
Oct 30 22:10:40.892174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.892152    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-openshift-ca\") pod \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") "
Oct 30 22:10:40.892306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.892286    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-kube-api-access-5wzjv\" (UniqueName: \"kubernetes.io/projected/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-kube-api-access-5wzjv\") pod \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\" (UID: \"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56\") "
Oct 30 22:10:40.892564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.892539    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-openshift-ca" pod "ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" (UID: "ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:40.893063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.893030    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:40.893925 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.893810    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-openshift-ca" pod "68a6f9ac-660e-4883-a565-9c8d172f77ef" (UID: "68a6f9ac-660e-4883-a565-9c8d172f77ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:40.895109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.895078    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-xks-ca" pod "ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" (UID: "ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:40.895199 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.895152    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-kube-api-access-5wzjv" pod "ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" (UID: "ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56"). InnerVolumeSpecName "kube-api-access-5wzjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:40.895869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.895840    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a6f9ac-660e-4883-a565-9c8d172f77ef-kube-api-access-rzz4m" pod "68a6f9ac-660e-4883-a565-9c8d172f77ef" (UID: "68a6f9ac-660e-4883-a565-9c8d172f77ef"). InnerVolumeSpecName "kube-api-access-rzz4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:40.896057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.896029    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-xks-ca" pod "68a6f9ac-660e-4883-a565-9c8d172f77ef" (UID: "68a6f9ac-660e-4883-a565-9c8d172f77ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:40.994763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.994721    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzjv\" (UniqueName: \"kubernetes.io/projected/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-kube-api-access-5wzjv\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:40.994763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.994760    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:40.994981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.994774    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:40.994981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.994790    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a6f9ac-660e-4883-a565-9c8d172f77ef-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:40.994981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:40.994802    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzz4m\" (UniqueName: \"kubernetes.io/projected/68a6f9ac-660e-4883-a565-9c8d172f77ef-kube-api-access-rzz4m\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:41.515636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.515596    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb" event={"ID":"ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56","Type":"ContainerDied","Data":"5c27ab98e8abc956ccc57464166e46277fad7f9ef6d1c4d24066e56e389e08c8"}
Oct 30 22:10:41.516092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.515656    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"
Oct 30 22:10:41.516092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.515680    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c27ab98e8abc956ccc57464166e46277fad7f9ef6d1c4d24066e56e389e08c8"
Oct 30 22:10:41.517423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.517396    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"
Oct 30 22:10:41.517540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.517439    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx" event={"ID":"68a6f9ac-660e-4883-a565-9c8d172f77ef","Type":"ContainerDied","Data":"12ddd20c1074f2fb61fde0880c83aeedef8af4689685fd224bb718d083be03ef"}
Oct 30 22:10:41.517540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.517475    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12ddd20c1074f2fb61fde0880c83aeedef8af4689685fd224bb718d083be03ef"
Oct 30 22:10:41.550664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.550447    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"]
Oct 30 22:10:41.551464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.551439    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" containerName="revoke-keys"
Oct 30 22:10:41.551464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.551464    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" containerName="revoke-keys"
Oct 30 22:10:41.551653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.551484    5031 cpu_manager.go:414] "containerMap: removing container" podUID="68a6f9ac-660e-4883-a565-9c8d172f77ef" containerName="revoke-keys"
Oct 30 22:10:41.551653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.551492    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a6f9ac-660e-4883-a565-9c8d172f77ef" containerName="revoke-keys"
Oct 30 22:10:41.551717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.551658    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" containerName="revoke-keys"
Oct 30 22:10:41.551717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.551668    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a6f9ac-660e-4883-a565-9c8d172f77ef" containerName="revoke-keys"
Oct 30 22:10:41.555705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.555682    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.604937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.604894    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqps\" (UniqueName: \"kubernetes.io/projected/5a10adbb-8b2d-4e02-b916-24143b74a75f-kube-api-access-lbqps\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.605104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.604995    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.605197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.605104    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.618814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.618784    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"]
Oct 30 22:10:41.694905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.694880    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:41.706034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.706002    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7cb0d7-3921-45ef-8d9c-11409121c349-kube-api-access-4p4hq\" (UniqueName: \"kubernetes.io/projected/fb7cb0d7-3921-45ef-8d9c-11409121c349-kube-api-access-4p4hq\") pod \"fb7cb0d7-3921-45ef-8d9c-11409121c349\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") "
Oct 30 22:10:41.706193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.706181    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-openshift-ca\") pod \"fb7cb0d7-3921-45ef-8d9c-11409121c349\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") "
Oct 30 22:10:41.706383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.706366    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-xks-ca\") pod \"fb7cb0d7-3921-45ef-8d9c-11409121c349\" (UID: \"fb7cb0d7-3921-45ef-8d9c-11409121c349\") "
Oct 30 22:10:41.706519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.706498    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-openshift-ca" pod "fb7cb0d7-3921-45ef-8d9c-11409121c349" (UID: "fb7cb0d7-3921-45ef-8d9c-11409121c349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:41.706593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.706580    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqps\" (UniqueName: \"kubernetes.io/projected/5a10adbb-8b2d-4e02-b916-24143b74a75f-kube-api-access-lbqps\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.706663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.706603    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.706715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.706670    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.707184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.707053    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:41.707367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.707340    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.708618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.708591    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7cb0d7-3921-45ef-8d9c-11409121c349-kube-api-access-4p4hq" pod "fb7cb0d7-3921-45ef-8d9c-11409121c349" (UID: "fb7cb0d7-3921-45ef-8d9c-11409121c349"). InnerVolumeSpecName "kube-api-access-4p4hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:41.708736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.708617    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-xks-ca" pod "fb7cb0d7-3921-45ef-8d9c-11409121c349" (UID: "fb7cb0d7-3921-45ef-8d9c-11409121c349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:41.709159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.709138    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.771360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.771285    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqps\" (UniqueName: \"kubernetes.io/projected/5a10adbb-8b2d-4e02-b916-24143b74a75f-kube-api-access-lbqps\") pod \"maas-api-revoke-sub-718ec46c-hgfnh\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:41.808930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.808897    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7cb0d7-3921-45ef-8d9c-11409121c349-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:41.808930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.808927    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p4hq\" (UniqueName: \"kubernetes.io/projected/fb7cb0d7-3921-45ef-8d9c-11409121c349-kube-api-access-4p4hq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:41.875113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:41.875080    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:42.285485 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:42.285441    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a10adbb_8b2d_4e02_b916_24143b74a75f.slice/crio-09c26d36c858e84b1f5936842984f08f75fff38015241430265c107bbb60a2db WatchSource:0}: Error finding container 09c26d36c858e84b1f5936842984f08f75fff38015241430265c107bbb60a2db: Status 404 returned error can't find the container with id 09c26d36c858e84b1f5936842984f08f75fff38015241430265c107bbb60a2db
Oct 30 22:10:42.286529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:42.286503    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"]
Oct 30 22:10:42.540893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:42.540798    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"
Oct 30 22:10:42.540893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:42.540810    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc" event={"ID":"fb7cb0d7-3921-45ef-8d9c-11409121c349","Type":"ContainerDied","Data":"92c3a3b0ce1dcce75dd9c8e77e023ca9b7720ece1d231eb5d6c54e05ccf745d9"}
Oct 30 22:10:42.540893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:42.540860    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c3a3b0ce1dcce75dd9c8e77e023ca9b7720ece1d231eb5d6c54e05ccf745d9"
Oct 30 22:10:42.544157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:42.544124    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh" event={"ID":"5a10adbb-8b2d-4e02-b916-24143b74a75f","Type":"ContainerStarted","Data":"09c26d36c858e84b1f5936842984f08f75fff38015241430265c107bbb60a2db"}
Oct 30 22:10:42.904202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:42.903671    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:43.060323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.060283    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"]
Oct 30 22:10:43.066156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.066125    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc4c407-t6jgj"]
Oct 30 22:10:43.357479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.357443    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"]
Oct 30 22:10:43.358574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.358552    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fb7cb0d7-3921-45ef-8d9c-11409121c349" containerName="revoke-keys"
Oct 30 22:10:43.358574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.358573    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7cb0d7-3921-45ef-8d9c-11409121c349" containerName="revoke-keys"
Oct 30 22:10:43.358795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.358783    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7cb0d7-3921-45ef-8d9c-11409121c349" containerName="revoke-keys"
Oct 30 22:10:43.363367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.363347    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.424421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.424386    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"]
Oct 30 22:10:43.436168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.436136    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.436913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.436386    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.436913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.436534    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7689c\" (UniqueName: \"kubernetes.io/projected/270eaa0b-341c-4a2f-aabc-f086dcba2acc-kube-api-access-7689c\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.538074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.538034    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.538285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.538163    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.538356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.538295    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7689c\" (UniqueName: \"kubernetes.io/projected/270eaa0b-341c-4a2f-aabc-f086dcba2acc-kube-api-access-7689c\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.538920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.538896    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.540774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.540751    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.558675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.558639    5031 generic.go:356] "Generic (PLEG): container finished" podID="5a10adbb-8b2d-4e02-b916-24143b74a75f" containerID="07a29c923156a9664ce400617b77e458436e35d87b632675eac62788e31cc34f" exitCode=0
Oct 30 22:10:43.559070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.558718    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh" event={"ID":"5a10adbb-8b2d-4e02-b916-24143b74a75f","Type":"ContainerDied","Data":"07a29c923156a9664ce400617b77e458436e35d87b632675eac62788e31cc34f"}
Oct 30 22:10:43.602905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.602868    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7689c\" (UniqueName: \"kubernetes.io/projected/270eaa0b-341c-4a2f-aabc-f086dcba2acc-kube-api-access-7689c\") pod \"maas-api-revoke-sub-c2b409a1-r2j6z\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:43.755606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:43.755524    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:44.148538 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:44.148506    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270eaa0b_341c_4a2f_aabc_f086dcba2acc.slice/crio-3c338b6e24b83cda365ff8283ffb0d7801c7585c29e335b76d57de248d4a6161 WatchSource:0}: Error finding container 3c338b6e24b83cda365ff8283ffb0d7801c7585c29e335b76d57de248d4a6161: Status 404 returned error can't find the container with id 3c338b6e24b83cda365ff8283ffb0d7801c7585c29e335b76d57de248d4a6161
Oct 30 22:10:44.148875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.148848    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"]
Oct 30 22:10:44.571517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.571474    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z" event={"ID":"270eaa0b-341c-4a2f-aabc-f086dcba2acc","Type":"ContainerStarted","Data":"8d485db4b90f4086dca65ccb1e8ba58d52deaeca19ac5c86fcb8106298253d32"}
Oct 30 22:10:44.572000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.571542    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z" event={"ID":"270eaa0b-341c-4a2f-aabc-f086dcba2acc","Type":"ContainerStarted","Data":"3c338b6e24b83cda365ff8283ffb0d7801c7585c29e335b76d57de248d4a6161"}
Oct 30 22:10:44.656238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.653855    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d033d7-48eb-4412-9a0d-5b9bbe392ccb" path="/var/lib/kubelet/pods/77d033d7-48eb-4412-9a0d-5b9bbe392ccb/volumes"
Oct 30 22:10:44.692193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.692060    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z" podStartSLOduration=1.692042667 podStartE2EDuration="1.692042667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:44.691625261 +0000 UTC m=+1256.215768046" watchObservedRunningTime="2026-10-30 22:10:44.692042667 +0000 UTC m=+1256.216185435"
Oct 30 22:10:44.737806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.737779    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:44.758183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.758150    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-xks-ca\") pod \"5a10adbb-8b2d-4e02-b916-24143b74a75f\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") "
Oct 30 22:10:44.758372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.758351    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-openshift-ca\") pod \"5a10adbb-8b2d-4e02-b916-24143b74a75f\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") "
Oct 30 22:10:44.758439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.758424    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a10adbb-8b2d-4e02-b916-24143b74a75f-kube-api-access-lbqps\" (UniqueName: \"kubernetes.io/projected/5a10adbb-8b2d-4e02-b916-24143b74a75f-kube-api-access-lbqps\") pod \"5a10adbb-8b2d-4e02-b916-24143b74a75f\" (UID: \"5a10adbb-8b2d-4e02-b916-24143b74a75f\") "
Oct 30 22:10:44.758766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.758734    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-openshift-ca" pod "5a10adbb-8b2d-4e02-b916-24143b74a75f" (UID: "5a10adbb-8b2d-4e02-b916-24143b74a75f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:44.759158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.759134    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.760508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.760484    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-xks-ca" pod "5a10adbb-8b2d-4e02-b916-24143b74a75f" (UID: "5a10adbb-8b2d-4e02-b916-24143b74a75f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:44.760606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.760589    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a10adbb-8b2d-4e02-b916-24143b74a75f-kube-api-access-lbqps" pod "5a10adbb-8b2d-4e02-b916-24143b74a75f" (UID: "5a10adbb-8b2d-4e02-b916-24143b74a75f"). InnerVolumeSpecName "kube-api-access-lbqps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.864799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.864763    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a10adbb-8b2d-4e02-b916-24143b74a75f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.865024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:44.865007    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqps\" (UniqueName: \"kubernetes.io/projected/5a10adbb-8b2d-4e02-b916-24143b74a75f-kube-api-access-lbqps\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.582433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:45.582335    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"
Oct 30 22:10:45.582433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:45.582339    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh" event={"ID":"5a10adbb-8b2d-4e02-b916-24143b74a75f","Type":"ContainerDied","Data":"09c26d36c858e84b1f5936842984f08f75fff38015241430265c107bbb60a2db"}
Oct 30 22:10:45.582907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:45.582506    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09c26d36c858e84b1f5936842984f08f75fff38015241430265c107bbb60a2db"
Oct 30 22:10:45.584385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:45.584356    5031 generic.go:356] "Generic (PLEG): container finished" podID="270eaa0b-341c-4a2f-aabc-f086dcba2acc" containerID="8d485db4b90f4086dca65ccb1e8ba58d52deaeca19ac5c86fcb8106298253d32" exitCode=0
Oct 30 22:10:45.584527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:45.584438    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z" event={"ID":"270eaa0b-341c-4a2f-aabc-f086dcba2acc","Type":"ContainerDied","Data":"8d485db4b90f4086dca65ccb1e8ba58d52deaeca19ac5c86fcb8106298253d32"}
Oct 30 22:10:45.806675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:45.806640    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:46.785035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.785009    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:46.897696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.897595    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270eaa0b-341c-4a2f-aabc-f086dcba2acc-kube-api-access-7689c\" (UniqueName: \"kubernetes.io/projected/270eaa0b-341c-4a2f-aabc-f086dcba2acc-kube-api-access-7689c\") pod \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") "
Oct 30 22:10:46.897866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.897774    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-xks-ca\") pod \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") "
Oct 30 22:10:46.897866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.897858    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-openshift-ca\") pod \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\" (UID: \"270eaa0b-341c-4a2f-aabc-f086dcba2acc\") "
Oct 30 22:10:46.898328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.898297    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-openshift-ca" pod "270eaa0b-341c-4a2f-aabc-f086dcba2acc" (UID: "270eaa0b-341c-4a2f-aabc-f086dcba2acc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:46.898980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.898953    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:46.900263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.900229    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270eaa0b-341c-4a2f-aabc-f086dcba2acc-kube-api-access-7689c" pod "270eaa0b-341c-4a2f-aabc-f086dcba2acc" (UID: "270eaa0b-341c-4a2f-aabc-f086dcba2acc"). InnerVolumeSpecName "kube-api-access-7689c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:46.900374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:46.900307    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-xks-ca" pod "270eaa0b-341c-4a2f-aabc-f086dcba2acc" (UID: "270eaa0b-341c-4a2f-aabc-f086dcba2acc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:47.001090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:47.001057    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270eaa0b-341c-4a2f-aabc-f086dcba2acc-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.001090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:47.001086    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7689c\" (UniqueName: \"kubernetes.io/projected/270eaa0b-341c-4a2f-aabc-f086dcba2acc-kube-api-access-7689c\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.606224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:47.606178    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"
Oct 30 22:10:47.606224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:47.606174    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z" event={"ID":"270eaa0b-341c-4a2f-aabc-f086dcba2acc","Type":"ContainerDied","Data":"3c338b6e24b83cda365ff8283ffb0d7801c7585c29e335b76d57de248d4a6161"}
Oct 30 22:10:47.606566 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:47.606336    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c338b6e24b83cda365ff8283ffb0d7801c7585c29e335b76d57de248d4a6161"
Oct 30 22:10:47.910649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:47.910550    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:49.077680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.077643    5031 scope.go:122] "RemoveContainer" containerID="5e618d910c16bc9a7d9904f0e3190851956d21e217fc0a30beffa74336a4f4ec"
Oct 30 22:10:49.089927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.089907    5031 scope.go:122] "RemoveContainer" containerID="fed45606bf5e1d9ca5d311c925395b2380911b8782de9f32a2ffb5bb70ca25bd"
Oct 30 22:10:49.101188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.101167    5031 scope.go:122] "RemoveContainer" containerID="0a4d61185a7ee3bc29f3b14e07c5b37095cf4766df914b7f3e9ab11716649a73"
Oct 30 22:10:49.115412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.115391    5031 scope.go:122] "RemoveContainer" containerID="2e80214dec4825ae13c8414e05667751a145b865c18f81265f796fd82e30ac3b"
Oct 30 22:10:49.125805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.125786    5031 scope.go:122] "RemoveContainer" containerID="6723413e528802fa7e3bb2d89b86d300dfbb2ba191b08c52dc3979b90691e940"
Oct 30 22:10:49.136563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.136542    5031 scope.go:122] "RemoveContainer" containerID="932c63de613e1e15f1f730f51c0b305e279b5a95ef2d32e72ae7cc6e8a937b56"
Oct 30 22:10:49.362887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.362854    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"]
Oct 30 22:10:49.363636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.363620    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5a10adbb-8b2d-4e02-b916-24143b74a75f" containerName="revoke-keys"
Oct 30 22:10:49.363695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.363638    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a10adbb-8b2d-4e02-b916-24143b74a75f" containerName="revoke-keys"
Oct 30 22:10:49.363695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.363664    5031 cpu_manager.go:414] "containerMap: removing container" podUID="270eaa0b-341c-4a2f-aabc-f086dcba2acc" containerName="revoke-keys"
Oct 30 22:10:49.363695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.363670    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270eaa0b-341c-4a2f-aabc-f086dcba2acc" containerName="revoke-keys"
Oct 30 22:10:49.363808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.363798    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="270eaa0b-341c-4a2f-aabc-f086dcba2acc" containerName="revoke-keys"
Oct 30 22:10:49.363841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.363819    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a10adbb-8b2d-4e02-b916-24143b74a75f" containerName="revoke-keys"
Oct 30 22:10:49.369763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.369742    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439295    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439443    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439613    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439687    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439750    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439787    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439849    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439907    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmxj\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-kube-api-access-kpmxj\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.439933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.439930    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541015    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541176    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541304    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541398    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541510    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541581    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541719    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541810    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmxj\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-kube-api-access-kpmxj\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541848    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.541949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.541893    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.542536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.542506    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.542684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.542652    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.543024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.542972    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.543295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.543264    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.545468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.545439    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.545831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.545808    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.584303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.584268    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"]
Oct 30 22:10:49.607557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.607522    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.607743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.607663    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmxj\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-kube-api-access-kpmxj\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:49.684749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:49.684661    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:50.110432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.110399    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:50.136184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.136148    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"]
Oct 30 22:10:50.136476 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:50.136437    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a70bb55_c72e_4dfd_9659_8de14b1448ff.slice/crio-f0b9d538d93b518be8686920383643632b2e04d723a75724232dc135bac5e1b7 WatchSource:0}: Error finding container f0b9d538d93b518be8686920383643632b2e04d723a75724232dc135bac5e1b7: Status 404 returned error can't find the container with id f0b9d538d93b518be8686920383643632b2e04d723a75724232dc135bac5e1b7
Oct 30 22:10:50.141015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.140970    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.141130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.141067    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.141176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.141120    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:10:50.548797 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:10:50.548695    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:10:50.648947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.648911    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" event={"ID":"2a70bb55-c72e-4dfd-9659-8de14b1448ff","Type":"ContainerStarted","Data":"313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773"}
Oct 30 22:10:50.649170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.648968    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" event={"ID":"2a70bb55-c72e-4dfd-9659-8de14b1448ff","Type":"ContainerStarted","Data":"f0b9d538d93b518be8686920383643632b2e04d723a75724232dc135bac5e1b7"}
Oct 30 22:10:50.684800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.684764    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:50.772906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:50.772849    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podStartSLOduration=1.772834222 podStartE2EDuration="1.772834222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:50.766819234 +0000 UTC m=+1262.290962009" watchObservedRunningTime="2026-10-30 22:10:50.772834222 +0000 UTC m=+1262.296976987"
Oct 30 22:10:51.058326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.058284    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"]
Oct 30 22:10:51.063239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.063195    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.121604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.121570    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:51.142234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.125348    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:51.142234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.126167    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:10:51.142234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.126305    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:51.155286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.153909    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"]
Oct 30 22:10:51.170732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.170697    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.170910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.170770    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.171027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.171008    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4pg\" (UniqueName: \"kubernetes.io/projected/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-kube-api-access-bm4pg\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.278232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.273899    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4pg\" (UniqueName: \"kubernetes.io/projected/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-kube-api-access-bm4pg\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.278232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.274718    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.278232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.275441    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.278232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.276279    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.280286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.279147    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.338455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.338378    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4pg\" (UniqueName: \"kubernetes.io/projected/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-kube-api-access-bm4pg\") pod \"maas-api-revoke-sub-a6204f20-frm8b\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.379780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.379746    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:51.687184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.687151    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:51.687385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.687251    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:51.805985 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:51.805954    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bd04ea_45c1_4f00_b8e7_f842c5ae3949.slice/crio-90c5544309a52654b643d8559db4b6cedae490e8dc300dfd1501309f271ca50e WatchSource:0}: Error finding container 90c5544309a52654b643d8559db4b6cedae490e8dc300dfd1501309f271ca50e: Status 404 returned error can't find the container with id 90c5544309a52654b643d8559db4b6cedae490e8dc300dfd1501309f271ca50e
Oct 30 22:10:51.822244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:51.819283    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"]
Oct 30 22:10:52.405806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.405769    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"]
Oct 30 22:10:52.439425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.438550    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.439425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.439360    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"]
Oct 30 22:10:52.446131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.446102    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.476360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.476301    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"]
Oct 30 22:10:52.494385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.494359    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.494599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.494584    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.494787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.494766    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.494952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.494928    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6skhl\" (UniqueName: \"kubernetes.io/projected/28f3490c-1f58-4451-9adb-19ed2dbae0b5-kube-api-access-6skhl\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.495020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.495003    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.495159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.495137    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67m7f\" (UniqueName: \"kubernetes.io/projected/423f0bc7-4a12-4d29-9a49-c2bc55b82016-kube-api-access-67m7f\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.550105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.550065    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"]
Oct 30 22:10:52.597827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.597701    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.598366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.598340    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.598795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.598774    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.599243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.599124    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6skhl\" (UniqueName: \"kubernetes.io/projected/28f3490c-1f58-4451-9adb-19ed2dbae0b5-kube-api-access-6skhl\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.599464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.599321    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.599667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.599423    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.599875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.599662    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.600118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.600099    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67m7f\" (UniqueName: \"kubernetes.io/projected/423f0bc7-4a12-4d29-9a49-c2bc55b82016-kube-api-access-67m7f\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.602171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.602153    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.602413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.602363    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.670086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.670001    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67m7f\" (UniqueName: \"kubernetes.io/projected/423f0bc7-4a12-4d29-9a49-c2bc55b82016-kube-api-access-67m7f\") pod \"maas-api-revoke-sub-b4705d6b-szkjd\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.670086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.670079    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6skhl\" (UniqueName: \"kubernetes.io/projected/28f3490c-1f58-4451-9adb-19ed2dbae0b5-kube-api-access-6skhl\") pod \"maas-api-revoke-sub-25d93752-v7tcf\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.685169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.685127    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:52.685381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.685192    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:52.691105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.691072    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b" event={"ID":"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949","Type":"ContainerStarted","Data":"ab0f45993c261bf67d006db13670c634e417cb21992264d84307bce05a0f10f0"}
Oct 30 22:10:52.691283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.691182    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b" event={"ID":"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949","Type":"ContainerStarted","Data":"90c5544309a52654b643d8559db4b6cedae490e8dc300dfd1501309f271ca50e"}
Oct 30 22:10:52.761687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.761654    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:52.780945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.779918    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:52.853322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:52.852590    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b" podStartSLOduration=1.852568253 podStartE2EDuration="1.852568253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:52.814640387 +0000 UTC m=+1264.338783153" watchObservedRunningTime="2026-10-30 22:10:52.852568253 +0000 UTC m=+1264.376711017"
Oct 30 22:10:53.191000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.188346    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"]
Oct 30 22:10:53.193509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.193250    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.198653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.198627    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"]
Oct 30 22:10:53.255961 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:53.255725    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423f0bc7_4a12_4d29_9a49_c2bc55b82016.slice/crio-1e84d6979903626565fec83570c53e550f0e31caea4d588c7216f12aa5ba7186 WatchSource:0}: Error finding container 1e84d6979903626565fec83570c53e550f0e31caea4d588c7216f12aa5ba7186: Status 404 returned error can't find the container with id 1e84d6979903626565fec83570c53e550f0e31caea4d588c7216f12aa5ba7186
Oct 30 22:10:53.262873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.262852    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"]
Oct 30 22:10:53.272776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.272751    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"]
Oct 30 22:10:53.288638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.288560    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:53.341015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.340364    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8phs\" (UniqueName: \"kubernetes.io/projected/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-kube-api-access-l8phs\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.341015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.340507    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.341015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.340607    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.451973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.451874    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8phs\" (UniqueName: \"kubernetes.io/projected/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-kube-api-access-l8phs\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.453484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.453191    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.453484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.453416    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.454822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.454416    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.525237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.521063    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.544365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.542123    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8phs\" (UniqueName: \"kubernetes.io/projected/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-kube-api-access-l8phs\") pod \"maas-api-revoke-sub-0d246c60-hg8h8\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:53.686628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.686596    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:53.686827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.686675    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:53.702286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.702120    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd" event={"ID":"423f0bc7-4a12-4d29-9a49-c2bc55b82016","Type":"ContainerStarted","Data":"1e84d6979903626565fec83570c53e550f0e31caea4d588c7216f12aa5ba7186"}
Oct 30 22:10:53.703672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.703645    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf" event={"ID":"28f3490c-1f58-4451-9adb-19ed2dbae0b5","Type":"ContainerStarted","Data":"e6dcd0fefb3937da9da76a3f5def7800aad1ff91d4bc9dcb3d45c7b7801d7ec7"}
Oct 30 22:10:53.705429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.705407    5031 generic.go:356] "Generic (PLEG): container finished" podID="f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" containerID="ab0f45993c261bf67d006db13670c634e417cb21992264d84307bce05a0f10f0" exitCode=0
Oct 30 22:10:53.705533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.705474    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b" event={"ID":"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949","Type":"ContainerDied","Data":"ab0f45993c261bf67d006db13670c634e417cb21992264d84307bce05a0f10f0"}
Oct 30 22:10:53.842112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:53.841857    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:54.267830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.267796    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"]
Oct 30 22:10:54.274581 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:54.272511    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa0e41e_b579_4ac6_ae1f_5e1995af6dc3.slice/crio-25ed06043302bb2e3255c76d01366fc12a85d41ae9ebfc1bb26b00fc399d6895 WatchSource:0}: Error finding container 25ed06043302bb2e3255c76d01366fc12a85d41ae9ebfc1bb26b00fc399d6895: Status 404 returned error can't find the container with id 25ed06043302bb2e3255c76d01366fc12a85d41ae9ebfc1bb26b00fc399d6895
Oct 30 22:10:54.684995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.684966    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:54.685460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.685433    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:54.746791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.746756    5031 generic.go:356] "Generic (PLEG): container finished" podID="423f0bc7-4a12-4d29-9a49-c2bc55b82016" containerID="daa7a7a73de46384f3e21a1a954d3e5a9d43e7eb6155647eb63a50fef6332554" exitCode=0
Oct 30 22:10:54.747620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.747590    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd" event={"ID":"423f0bc7-4a12-4d29-9a49-c2bc55b82016","Type":"ContainerDied","Data":"daa7a7a73de46384f3e21a1a954d3e5a9d43e7eb6155647eb63a50fef6332554"}
Oct 30 22:10:54.749602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.749574    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8" event={"ID":"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3","Type":"ContainerStarted","Data":"25ed06043302bb2e3255c76d01366fc12a85d41ae9ebfc1bb26b00fc399d6895"}
Oct 30 22:10:54.751876 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.751851    5031 generic.go:356] "Generic (PLEG): container finished" podID="28f3490c-1f58-4451-9adb-19ed2dbae0b5" containerID="41ad170d3a5df11b64084bad3c843df8e665b35569f703939c1fd370d6c491e6" exitCode=0
Oct 30 22:10:54.752388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:54.752027    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf" event={"ID":"28f3490c-1f58-4451-9adb-19ed2dbae0b5","Type":"ContainerDied","Data":"41ad170d3a5df11b64084bad3c843df8e665b35569f703939c1fd370d6c491e6"}
Oct 30 22:10:55.147334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.147190    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:55.282742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.282700    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-xks-ca\") pod \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") "
Oct 30 22:10:55.282955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.282799    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-openshift-ca\") pod \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") "
Oct 30 22:10:55.283094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.283071    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-kube-api-access-bm4pg\" (UniqueName: \"kubernetes.io/projected/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-kube-api-access-bm4pg\") pod \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\" (UID: \"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949\") "
Oct 30 22:10:55.283966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.283940    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-openshift-ca" pod "f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" (UID: "f2bd04ea-45c1-4f00-b8e7-f842c5ae3949"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:55.286329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.286305    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-kube-api-access-bm4pg" pod "f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" (UID: "f2bd04ea-45c1-4f00-b8e7-f842c5ae3949"). InnerVolumeSpecName "kube-api-access-bm4pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:55.286736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.286711    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-xks-ca" pod "f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" (UID: "f2bd04ea-45c1-4f00-b8e7-f842c5ae3949"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:55.385512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.385475    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm4pg\" (UniqueName: \"kubernetes.io/projected/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-kube-api-access-bm4pg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:55.385802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.385778    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:55.385969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.385953    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:55.685418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.685375    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:55.685929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.685443    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:55.782124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.779878    5031 generic.go:356] "Generic (PLEG): container finished" podID="eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" containerID="961d1ce8a63ffde351fb3b58ae19cb18f0a163632248bbe4d17d33ff04916e15" exitCode=0
Oct 30 22:10:55.782124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.780057    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8" event={"ID":"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3","Type":"ContainerDied","Data":"961d1ce8a63ffde351fb3b58ae19cb18f0a163632248bbe4d17d33ff04916e15"}
Oct 30 22:10:55.782576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.782548    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"
Oct 30 22:10:55.782856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.782817    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b" event={"ID":"f2bd04ea-45c1-4f00-b8e7-f842c5ae3949","Type":"ContainerDied","Data":"90c5544309a52654b643d8559db4b6cedae490e8dc300dfd1501309f271ca50e"}
Oct 30 22:10:55.782976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:55.782879    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90c5544309a52654b643d8559db4b6cedae490e8dc300dfd1501309f271ca50e"
Oct 30 22:10:56.166463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.166429    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:56.200119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.200088    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423f0bc7-4a12-4d29-9a49-c2bc55b82016-kube-api-access-67m7f\" (UniqueName: \"kubernetes.io/projected/423f0bc7-4a12-4d29-9a49-c2bc55b82016-kube-api-access-67m7f\") pod \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") "
Oct 30 22:10:56.200330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.200144    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-openshift-ca\") pod \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") "
Oct 30 22:10:56.200330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.200310    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-xks-ca\") pod \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\" (UID: \"423f0bc7-4a12-4d29-9a49-c2bc55b82016\") "
Oct 30 22:10:56.200992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.200735    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-openshift-ca" pod "423f0bc7-4a12-4d29-9a49-c2bc55b82016" (UID: "423f0bc7-4a12-4d29-9a49-c2bc55b82016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:56.201112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.201089    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:56.206142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.206109    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423f0bc7-4a12-4d29-9a49-c2bc55b82016-kube-api-access-67m7f" pod "423f0bc7-4a12-4d29-9a49-c2bc55b82016" (UID: "423f0bc7-4a12-4d29-9a49-c2bc55b82016"). InnerVolumeSpecName "kube-api-access-67m7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:56.240056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.240005    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-xks-ca" pod "423f0bc7-4a12-4d29-9a49-c2bc55b82016" (UID: "423f0bc7-4a12-4d29-9a49-c2bc55b82016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:56.264340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.264310    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:56.307230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.301936    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f3490c-1f58-4451-9adb-19ed2dbae0b5-kube-api-access-6skhl\" (UniqueName: \"kubernetes.io/projected/28f3490c-1f58-4451-9adb-19ed2dbae0b5-kube-api-access-6skhl\") pod \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") "
Oct 30 22:10:56.307230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.302128    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-openshift-ca\") pod \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") "
Oct 30 22:10:56.307230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.302185    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-xks-ca\") pod \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\" (UID: \"28f3490c-1f58-4451-9adb-19ed2dbae0b5\") "
Oct 30 22:10:56.307230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.302945    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423f0bc7-4a12-4d29-9a49-c2bc55b82016-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:56.307230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.302967    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67m7f\" (UniqueName: \"kubernetes.io/projected/423f0bc7-4a12-4d29-9a49-c2bc55b82016-kube-api-access-67m7f\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:56.307230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.303087    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-openshift-ca" pod "28f3490c-1f58-4451-9adb-19ed2dbae0b5" (UID: "28f3490c-1f58-4451-9adb-19ed2dbae0b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:56.311174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.311080    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f3490c-1f58-4451-9adb-19ed2dbae0b5-kube-api-access-6skhl" pod "28f3490c-1f58-4451-9adb-19ed2dbae0b5" (UID: "28f3490c-1f58-4451-9adb-19ed2dbae0b5"). InnerVolumeSpecName "kube-api-access-6skhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:56.313472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.313429    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-xks-ca" pod "28f3490c-1f58-4451-9adb-19ed2dbae0b5" (UID: "28f3490c-1f58-4451-9adb-19ed2dbae0b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:56.404240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.404185    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6skhl\" (UniqueName: \"kubernetes.io/projected/28f3490c-1f58-4451-9adb-19ed2dbae0b5-kube-api-access-6skhl\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:56.404240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.404240    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:56.404454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.404254    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f3490c-1f58-4451-9adb-19ed2dbae0b5-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:56.546079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.546040    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"]
Oct 30 22:10:56.546905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.546884    5031 cpu_manager.go:414] "containerMap: removing container" podUID="423f0bc7-4a12-4d29-9a49-c2bc55b82016" containerName="revoke-keys"
Oct 30 22:10:56.547028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.546906    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423f0bc7-4a12-4d29-9a49-c2bc55b82016" containerName="revoke-keys"
Oct 30 22:10:56.547028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.546920    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" containerName="revoke-keys"
Oct 30 22:10:56.547028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.546927    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" containerName="revoke-keys"
Oct 30 22:10:56.547028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.546984    5031 cpu_manager.go:414] "containerMap: removing container" podUID="28f3490c-1f58-4451-9adb-19ed2dbae0b5" containerName="revoke-keys"
Oct 30 22:10:56.547028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.546992    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f3490c-1f58-4451-9adb-19ed2dbae0b5" containerName="revoke-keys"
Oct 30 22:10:56.547305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.547180    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="423f0bc7-4a12-4d29-9a49-c2bc55b82016" containerName="revoke-keys"
Oct 30 22:10:56.547305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.547195    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" containerName="revoke-keys"
Oct 30 22:10:56.547305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.547235    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f3490c-1f58-4451-9adb-19ed2dbae0b5" containerName="revoke-keys"
Oct 30 22:10:56.551225 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.551189    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.608811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.608760    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.609024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.608831    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxg9\" (UniqueName: \"kubernetes.io/projected/eef016b2-cb83-424d-9f97-06f6bdb78068-kube-api-access-xlxg9\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.609024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.608931    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.618348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.618284    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"]
Oct 30 22:10:56.685639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.685607    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:56.686102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.685665    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:56.710135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.710093    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.710696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.710671    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.710814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.710722    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxg9\" (UniqueName: \"kubernetes.io/projected/eef016b2-cb83-424d-9f97-06f6bdb78068-kube-api-access-xlxg9\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.711092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.711069    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.714418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.714347    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.775988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.775953    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxg9\" (UniqueName: \"kubernetes.io/projected/eef016b2-cb83-424d-9f97-06f6bdb78068-kube-api-access-xlxg9\") pod \"maas-api-revoke-sub-015e1563-2882g\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:56.794526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.794493    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"
Oct 30 22:10:56.794731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.794529    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf" event={"ID":"28f3490c-1f58-4451-9adb-19ed2dbae0b5","Type":"ContainerDied","Data":"e6dcd0fefb3937da9da76a3f5def7800aad1ff91d4bc9dcb3d45c7b7801d7ec7"}
Oct 30 22:10:56.794731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.794589    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6dcd0fefb3937da9da76a3f5def7800aad1ff91d4bc9dcb3d45c7b7801d7ec7"
Oct 30 22:10:56.796488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.796465    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"
Oct 30 22:10:56.796607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.796516    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd" event={"ID":"423f0bc7-4a12-4d29-9a49-c2bc55b82016","Type":"ContainerDied","Data":"1e84d6979903626565fec83570c53e550f0e31caea4d588c7216f12aa5ba7186"}
Oct 30 22:10:56.796607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.796554    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e84d6979903626565fec83570c53e550f0e31caea4d588c7216f12aa5ba7186"
Oct 30 22:10:56.868291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:56.868189    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:10:57.064547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.064512    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:57.124563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.124450    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-openshift-ca\") pod \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") "
Oct 30 22:10:57.124754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.124634    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-kube-api-access-l8phs\" (UniqueName: \"kubernetes.io/projected/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-kube-api-access-l8phs\") pod \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") "
Oct 30 22:10:57.124813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.124771    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-xks-ca\") pod \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\" (UID: \"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3\") "
Oct 30 22:10:57.125170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.125140    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-openshift-ca" pod "eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" (UID: "eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:57.125557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.125533    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:57.128391 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.128363    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-xks-ca" pod "eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" (UID: "eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:57.129177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.129144    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-kube-api-access-l8phs" pod "eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" (UID: "eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3"). InnerVolumeSpecName "kube-api-access-l8phs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:57.243482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.243436    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8phs\" (UniqueName: \"kubernetes.io/projected/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-kube-api-access-l8phs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:57.243482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.243492    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:57.334594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.333989    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"]
Oct 30 22:10:57.819593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.819554    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8" event={"ID":"eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3","Type":"ContainerDied","Data":"25ed06043302bb2e3255c76d01366fc12a85d41ae9ebfc1bb26b00fc399d6895"}
Oct 30 22:10:57.820274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.820251    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ed06043302bb2e3255c76d01366fc12a85d41ae9ebfc1bb26b00fc399d6895"
Oct 30 22:10:57.820462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.820166    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"
Oct 30 22:10:57.829043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.829010    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g" event={"ID":"eef016b2-cb83-424d-9f97-06f6bdb78068","Type":"ContainerStarted","Data":"8d89a3339c603a469bd3bc48ea8ac70667cffb06ef5e40aba3ad8819f34c0f77"}
Oct 30 22:10:57.915891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.914646    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:57.916095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.916041    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:57.916839 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:57.916803    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:10:58.254431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.254392    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"]
Oct 30 22:10:58.255810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.255779    5031 cpu_manager.go:414] "containerMap: removing container" podUID="eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" containerName="revoke-keys"
Oct 30 22:10:58.256046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.256027    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" containerName="revoke-keys"
Oct 30 22:10:58.256400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.256382    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" containerName="revoke-keys"
Oct 30 22:10:58.261661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.261417    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.335754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.335702    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"]
Oct 30 22:10:58.365702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.365670    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.365920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.365732    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.365920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.365758    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2bs\" (UniqueName: \"kubernetes.io/projected/2f63ec52-3309-4a43-bd73-bb7da69b1f19-kube-api-access-2j2bs\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.467565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.467519    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.467782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.467610    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.467782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.467641    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2bs\" (UniqueName: \"kubernetes.io/projected/2f63ec52-3309-4a43-bd73-bb7da69b1f19-kube-api-access-2j2bs\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.469364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.468935    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.494230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.494104    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.516530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.513049    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.544099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.541687    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2bs\" (UniqueName: \"kubernetes.io/projected/2f63ec52-3309-4a43-bd73-bb7da69b1f19-kube-api-access-2j2bs\") pod \"maas-api-revoke-sub-41214da0-bssz4\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.579986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.579945    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:10:58.841196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.841097    5031 generic.go:356] "Generic (PLEG): container finished" podID="eef016b2-cb83-424d-9f97-06f6bdb78068" containerID="2ea0c7d34943133100f46063ec03da5ff4de129f54417c79017d804f6b6aa404" exitCode=0
Oct 30 22:10:58.841685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.841185    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g" event={"ID":"eef016b2-cb83-424d-9f97-06f6bdb78068","Type":"ContainerDied","Data":"2ea0c7d34943133100f46063ec03da5ff4de129f54417c79017d804f6b6aa404"}
Oct 30 22:10:58.997147 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:10:58.997117    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f63ec52_3309_4a43_bd73_bb7da69b1f19.slice/crio-2eb2e6651543c79ae44aa6bcb91563fcf89fa44811aea0fb39934253e21811c2 WatchSource:0}: Error finding container 2eb2e6651543c79ae44aa6bcb91563fcf89fa44811aea0fb39934253e21811c2: Status 404 returned error can't find the container with id 2eb2e6651543c79ae44aa6bcb91563fcf89fa44811aea0fb39934253e21811c2
Oct 30 22:10:59.000051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:58.999973    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"]
Oct 30 22:10:59.113877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:59.113840    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"]
Oct 30 22:10:59.122944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:59.122911    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ffe514-zxbk2"]
Oct 30 22:10:59.858429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:59.858393    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4" event={"ID":"2f63ec52-3309-4a43-bd73-bb7da69b1f19","Type":"ContainerStarted","Data":"7719430ea982ad531d879b469a575ead267cf89296fba050cd64adde15bb772e"}
Oct 30 22:10:59.858976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:59.858953    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4" event={"ID":"2f63ec52-3309-4a43-bd73-bb7da69b1f19","Type":"ContainerStarted","Data":"2eb2e6651543c79ae44aa6bcb91563fcf89fa44811aea0fb39934253e21811c2"}
Oct 30 22:10:59.981379 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:10:59.981312    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4" podStartSLOduration=1.981292985 podStartE2EDuration="1.981292985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:59.980983861 +0000 UTC m=+1271.505126626" watchObservedRunningTime="2026-10-30 22:10:59.981292985 +0000 UTC m=+1271.505435749"
Oct 30 22:11:00.095126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.095097    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:11:00.097683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.097662    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-xks-ca\") pod \"eef016b2-cb83-424d-9f97-06f6bdb78068\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") "
Oct 30 22:11:00.097794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.097724    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-openshift-ca\") pod \"eef016b2-cb83-424d-9f97-06f6bdb78068\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") "
Oct 30 22:11:00.097874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.097863    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef016b2-cb83-424d-9f97-06f6bdb78068-kube-api-access-xlxg9\" (UniqueName: \"kubernetes.io/projected/eef016b2-cb83-424d-9f97-06f6bdb78068-kube-api-access-xlxg9\") pod \"eef016b2-cb83-424d-9f97-06f6bdb78068\" (UID: \"eef016b2-cb83-424d-9f97-06f6bdb78068\") "
Oct 30 22:11:00.098116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.098089    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-openshift-ca" pod "eef016b2-cb83-424d-9f97-06f6bdb78068" (UID: "eef016b2-cb83-424d-9f97-06f6bdb78068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:00.098470 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.098453    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.100114 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.100093    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-xks-ca" pod "eef016b2-cb83-424d-9f97-06f6bdb78068" (UID: "eef016b2-cb83-424d-9f97-06f6bdb78068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:00.100244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.100122    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef016b2-cb83-424d-9f97-06f6bdb78068-kube-api-access-xlxg9" pod "eef016b2-cb83-424d-9f97-06f6bdb78068" (UID: "eef016b2-cb83-424d-9f97-06f6bdb78068"). InnerVolumeSpecName "kube-api-access-xlxg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:00.200435 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.200402    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef016b2-cb83-424d-9f97-06f6bdb78068-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.200435 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.200432    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlxg9\" (UniqueName: \"kubernetes.io/projected/eef016b2-cb83-424d-9f97-06f6bdb78068-kube-api-access-xlxg9\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.384026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.383936    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"]
Oct 30 22:11:00.384765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.384749    5031 cpu_manager.go:414] "containerMap: removing container" podUID="eef016b2-cb83-424d-9f97-06f6bdb78068" containerName="revoke-keys"
Oct 30 22:11:00.384844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.384767    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef016b2-cb83-424d-9f97-06f6bdb78068" containerName="revoke-keys"
Oct 30 22:11:00.384922 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.384912    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef016b2-cb83-424d-9f97-06f6bdb78068" containerName="revoke-keys"
Oct 30 22:11:00.389077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.389052    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.403131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.403091    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkhg\" (UniqueName: \"kubernetes.io/projected/00b02994-fb3b-4d41-95e1-06ad8e06a667-kube-api-access-hvkhg\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.403279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.403178    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.403400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.403378    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.449915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.449881    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"]
Oct 30 22:11:00.504905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.504865    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkhg\" (UniqueName: \"kubernetes.io/projected/00b02994-fb3b-4d41-95e1-06ad8e06a667-kube-api-access-hvkhg\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.505096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.504954    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.505096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.505079    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.505848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.505816    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.515894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.515858    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.569356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.569322    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkhg\" (UniqueName: \"kubernetes.io/projected/00b02994-fb3b-4d41-95e1-06ad8e06a667-kube-api-access-hvkhg\") pod \"maas-api-revoke-sub-ace4b013-6dn5w\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.649916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.649828    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3017489f-4cbb-4f8a-b5a8-4b954e78db1b" path="/var/lib/kubelet/pods/3017489f-4cbb-4f8a-b5a8-4b954e78db1b/volumes"
Oct 30 22:11:00.705726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.705692    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:00.872077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.872045    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"
Oct 30 22:11:00.872598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.872079    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g" event={"ID":"eef016b2-cb83-424d-9f97-06f6bdb78068","Type":"ContainerDied","Data":"8d89a3339c603a469bd3bc48ea8ac70667cffb06ef5e40aba3ad8819f34c0f77"}
Oct 30 22:11:00.872598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.872140    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d89a3339c603a469bd3bc48ea8ac70667cffb06ef5e40aba3ad8819f34c0f77"
Oct 30 22:11:00.874772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.874743    5031 generic.go:356] "Generic (PLEG): container finished" podID="2f63ec52-3309-4a43-bd73-bb7da69b1f19" containerID="7719430ea982ad531d879b469a575ead267cf89296fba050cd64adde15bb772e" exitCode=0
Oct 30 22:11:00.874914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:00.874783    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4" event={"ID":"2f63ec52-3309-4a43-bd73-bb7da69b1f19","Type":"ContainerDied","Data":"7719430ea982ad531d879b469a575ead267cf89296fba050cd64adde15bb772e"}
Oct 30 22:11:01.117442 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:01.117397    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b02994_fb3b_4d41_95e1_06ad8e06a667.slice/crio-022dcd9390ce5381118625f403799252dd197dab41ceda7341958dae69256598 WatchSource:0}: Error finding container 022dcd9390ce5381118625f403799252dd197dab41ceda7341958dae69256598: Status 404 returned error can't find the container with id 022dcd9390ce5381118625f403799252dd197dab41ceda7341958dae69256598
Oct 30 22:11:01.118824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:01.118795    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"]
Oct 30 22:11:01.886567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:01.886530    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w" event={"ID":"00b02994-fb3b-4d41-95e1-06ad8e06a667","Type":"ContainerStarted","Data":"5c016a0716fb16e6beceb30b1d5a20827fa0d2a23f1df3dda83d23ba523d4ec8"}
Oct 30 22:11:01.887043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:01.886588    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w" event={"ID":"00b02994-fb3b-4d41-95e1-06ad8e06a667","Type":"ContainerStarted","Data":"022dcd9390ce5381118625f403799252dd197dab41ceda7341958dae69256598"}
Oct 30 22:11:02.021609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.021265    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w" podStartSLOduration=2.021243069 podStartE2EDuration="2.021243069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:02.01119808 +0000 UTC m=+1273.535340845" watchObservedRunningTime="2026-10-30 22:11:02.021243069 +0000 UTC m=+1273.545385834"
Oct 30 22:11:02.095429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.095401    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:11:02.134443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.134403    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f63ec52-3309-4a43-bd73-bb7da69b1f19-kube-api-access-2j2bs\" (UniqueName: \"kubernetes.io/projected/2f63ec52-3309-4a43-bd73-bb7da69b1f19-kube-api-access-2j2bs\") pod \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") "
Oct 30 22:11:02.134657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.134542    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-openshift-ca\") pod \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") "
Oct 30 22:11:02.134657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.134569    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-xks-ca\") pod \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\" (UID: \"2f63ec52-3309-4a43-bd73-bb7da69b1f19\") "
Oct 30 22:11:02.135009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.134979    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-openshift-ca" pod "2f63ec52-3309-4a43-bd73-bb7da69b1f19" (UID: "2f63ec52-3309-4a43-bd73-bb7da69b1f19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:02.135152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.135131    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.136973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.136914    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-xks-ca" pod "2f63ec52-3309-4a43-bd73-bb7da69b1f19" (UID: "2f63ec52-3309-4a43-bd73-bb7da69b1f19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:02.137066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.136968    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f63ec52-3309-4a43-bd73-bb7da69b1f19-kube-api-access-2j2bs" pod "2f63ec52-3309-4a43-bd73-bb7da69b1f19" (UID: "2f63ec52-3309-4a43-bd73-bb7da69b1f19"). InnerVolumeSpecName "kube-api-access-2j2bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:02.237114 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.237070    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2bs\" (UniqueName: \"kubernetes.io/projected/2f63ec52-3309-4a43-bd73-bb7da69b1f19-kube-api-access-2j2bs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.237114 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.237118    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f63ec52-3309-4a43-bd73-bb7da69b1f19-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.928491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.928389    5031 generic.go:356] "Generic (PLEG): container finished" podID="00b02994-fb3b-4d41-95e1-06ad8e06a667" containerID="5c016a0716fb16e6beceb30b1d5a20827fa0d2a23f1df3dda83d23ba523d4ec8" exitCode=0
Oct 30 22:11:02.929749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.929527    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w" event={"ID":"00b02994-fb3b-4d41-95e1-06ad8e06a667","Type":"ContainerDied","Data":"5c016a0716fb16e6beceb30b1d5a20827fa0d2a23f1df3dda83d23ba523d4ec8"}
Oct 30 22:11:02.946421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.946382    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4" event={"ID":"2f63ec52-3309-4a43-bd73-bb7da69b1f19","Type":"ContainerDied","Data":"2eb2e6651543c79ae44aa6bcb91563fcf89fa44811aea0fb39934253e21811c2"}
Oct 30 22:11:02.946785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.946768    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb2e6651543c79ae44aa6bcb91563fcf89fa44811aea0fb39934253e21811c2"
Oct 30 22:11:02.946892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:02.946718    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"
Oct 30 22:11:03.904756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:03.904720    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:04.380238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:04.371025    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.027239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.026423    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.396882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.334306    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.396882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.358644    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.396882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.388863    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.445450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.415863    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:11:05.520239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.470821    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.654581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.639317    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:05.830236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.265433    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:11:05.967472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:05.967339    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.008506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.008363    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:11:06.040490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.027602    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:11:06.443195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.443144    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.482194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.482139    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.482794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.449584    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:11:06.515237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.483408    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:11:06.685689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.685290    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:11:06.695320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.491655    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.704234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.646876    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:06.811351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.810728    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:07.029311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.898517    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:11:07.029311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.899963    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-xks-ca\") pod \"00b02994-fb3b-4d41-95e1-06ad8e06a667\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") "
Oct 30 22:11:07.029311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.900042    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b02994-fb3b-4d41-95e1-06ad8e06a667-kube-api-access-hvkhg\" (UniqueName: \"kubernetes.io/projected/00b02994-fb3b-4d41-95e1-06ad8e06a667-kube-api-access-hvkhg\") pod \"00b02994-fb3b-4d41-95e1-06ad8e06a667\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") "
Oct 30 22:11:07.029311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:06.900158    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-openshift-ca\") pod \"00b02994-fb3b-4d41-95e1-06ad8e06a667\" (UID: \"00b02994-fb3b-4d41-95e1-06ad8e06a667\") "
Oct 30 22:11:07.029311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.021728    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:11:07.107265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.092570    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.107265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.092801    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.107265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.093282    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:07.271006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.215685    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-xks-ca" pod "00b02994-fb3b-4d41-95e1-06ad8e06a667" (UID: "00b02994-fb3b-4d41-95e1-06ad8e06a667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:07.578749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.578684    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-openshift-ca" pod "00b02994-fb3b-4d41-95e1-06ad8e06a667" (UID: "00b02994-fb3b-4d41-95e1-06ad8e06a667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:07.764716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.714709    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b02994-fb3b-4d41-95e1-06ad8e06a667-kube-api-access-hvkhg" pod "00b02994-fb3b-4d41-95e1-06ad8e06a667" (UID: "00b02994-fb3b-4d41-95e1-06ad8e06a667"). InnerVolumeSpecName "kube-api-access-hvkhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:07.764952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.597003    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.281624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.597256    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.282088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.714180    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded"
Oct 30 22:11:08.289355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.740188    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.289735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.756366    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.307769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.769455    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:08.308098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.308043    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkhg\" (UniqueName: \"kubernetes.io/projected/00b02994-fb3b-4d41-95e1-06ad8e06a667-kube-api-access-hvkhg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:08.308325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.308308    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b02994-fb3b-4d41-95e1-06ad8e06a667-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:08.308452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.166081    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.308771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.596687    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m" podUID="ed248874-db33-429b-b509-2a75e3e72304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.309112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:07.884833    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.427192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.197759    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m" podUID="ed248874-db33-429b-b509-2a75e3e72304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.427643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.259845    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.427962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.400926    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"
Oct 30 22:11:08.537703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.537542    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:08.538175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.538030    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:11:08.676255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:08.676070    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:11:09.227452 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:09.227412    5031 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.542s"
Oct 30 22:11:09.310234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.309120    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.310234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.309460    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.310596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.310571    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:11:09.311734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.311703    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:11:09.376570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.376527    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.453286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.417311    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:09.453286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.417390    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.453286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.419427    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.453286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.445983    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.541853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.541378    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.733291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.723406    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:09.839881 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.839757    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w" event={"ID":"00b02994-fb3b-4d41-95e1-06ad8e06a667","Type":"ContainerDied","Data":"022dcd9390ce5381118625f403799252dd197dab41ceda7341958dae69256598"}
Oct 30 22:11:09.908479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.908443    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="022dcd9390ce5381118625f403799252dd197dab41ceda7341958dae69256598"
Oct 30 22:11:09.933243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.910108    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:11:09.978476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.978434    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:11:09.978806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.978791    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:11:09.978978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.978966    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:11:09.979119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.979108    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:11:09.979281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:09.979269    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:11:10.282773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.282740    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"]
Oct 30 22:11:10.366928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.355195    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.369620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.369575    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.375674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.372097    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.391265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.385185    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.428681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.428622    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.464237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.440285    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.469683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.469634    5031 cpu_manager.go:414] "containerMap: removing container" podUID="00b02994-fb3b-4d41-95e1-06ad8e06a667" containerName="revoke-keys"
Oct 30 22:11:10.469907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.469888    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b02994-fb3b-4d41-95e1-06ad8e06a667" containerName="revoke-keys"
Oct 30 22:11:10.470080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.470066    5031 cpu_manager.go:414] "containerMap: removing container" podUID="2f63ec52-3309-4a43-bd73-bb7da69b1f19" containerName="revoke-keys"
Oct 30 22:11:10.483272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.470182    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f63ec52-3309-4a43-bd73-bb7da69b1f19" containerName="revoke-keys"
Oct 30 22:11:10.497463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.497430    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b02994-fb3b-4d41-95e1-06ad8e06a667" containerName="revoke-keys"
Oct 30 22:11:10.497837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.497775    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f63ec52-3309-4a43-bd73-bb7da69b1f19" containerName="revoke-keys"
Oct 30 22:11:10.747231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.712946    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:11:10.793777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.786323    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.808317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.794471    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:10.808317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.794519    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.808317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.795311    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:10.930369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.930321    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:10.945062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.930422    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:11:10.961682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.930476    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:10.961900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:10.930527    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:11.085419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.039661    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:11.085419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.052678    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.085419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.076883    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.085419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.077053    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfvc\" (UniqueName: \"kubernetes.io/projected/a7c2273c-603f-402f-bbb9-fbe262767807-kube-api-access-wzfvc\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.216609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.205734    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"]
Oct 30 22:11:11.280301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.276040    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.280762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.280714    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.321019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.292474    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.322581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.321599    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfvc\" (UniqueName: \"kubernetes.io/projected/a7c2273c-603f-402f-bbb9-fbe262767807-kube-api-access-wzfvc\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.451333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.338722    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.451333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.369490    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6923-e49089"]
Oct 30 22:11:11.451333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.405603    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:11.451333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.406009    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:11.451333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.406318    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:11:11.451333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.406563    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:11:11.451333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.406819    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn" podUID="8a6d5c6a-ec67-4875-960c-916306762084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 22:11:11.457266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.453366    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:11:11.508253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.489081    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:11.546990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.546945    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:11.548750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.548723    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:11.562527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.562484    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:11.657255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.634442    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9tw55\""
Oct 30 22:11:11.783035 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:11.782945    5031 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b02994_fb3b_4d41_95e1_06ad8e06a667.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:11:11.878916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.875895    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfvc\" (UniqueName: \"kubernetes.io/projected/a7c2273c-603f-402f-bbb9-fbe262767807-kube-api-access-wzfvc\") pod \"maas-api-revoke-sub-d390cf28-nhctv\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.893273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.889898    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:11:11.903259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.897701    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:11:11.903259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.902332    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:11:11.934232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.924704    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:11.934232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.928167    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wbc\" (UniqueName: \"kubernetes.io/projected/fa1d9c60-0788-46d7-8ef5-d833caf75a6b-kube-api-access-n9wbc\") pod \"test-curl-6923-e49089\" (UID: \"fa1d9c60-0788-46d7-8ef5-d833caf75a6b\") " pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:11.934232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.931617    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6923-e49089"]
Oct 30 22:11:11.944282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.944199    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:11.959978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.954401    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:11:11.959978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.954498    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:11:11.970237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:11.968399    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:11:12.020309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.018521    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:11:12.020309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.018797    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:11:12.020730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.020634    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6908-5e2e20"]
Oct 30 22:11:12.033463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.032422    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wbc\" (UniqueName: \"kubernetes.io/projected/fa1d9c60-0788-46d7-8ef5-d833caf75a6b-kube-api-access-n9wbc\") pod \"test-curl-6923-e49089\" (UID: \"fa1d9c60-0788-46d7-8ef5-d833caf75a6b\") " pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:12.036465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.036433    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:11:12.036609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.036488    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:11:12.037222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.037181    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:12.047939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.040372    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-5e2e20"]
Oct 30 22:11:12.059023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.058993    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"]
Oct 30 22:11:12.064676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.064648    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.065580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.065516    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"]
Oct 30 22:11:12.097545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.097515    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wbc\" (UniqueName: \"kubernetes.io/projected/fa1d9c60-0788-46d7-8ef5-d833caf75a6b-kube-api-access-n9wbc\") pod \"test-curl-6923-e49089\" (UID: \"fa1d9c60-0788-46d7-8ef5-d833caf75a6b\") " pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:12.099173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.099150    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"]
Oct 30 22:11:12.124663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.122815    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.124663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.123078    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"]
Oct 30 22:11:12.136636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.135434    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84x77\" (UniqueName: \"kubernetes.io/projected/69c72c5c-5abe-4297-aa72-8679b1c3d042-kube-api-access-84x77\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.136636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.135690    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.136636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.135915    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8xp\" (UniqueName: \"kubernetes.io/projected/fa4d1827-68ff-4e1b-951a-1d37bc62c062-kube-api-access-vd8xp\") pod \"test-curl-6908-5e2e20\" (UID: \"fa4d1827-68ff-4e1b-951a-1d37bc62c062\") " pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:12.136636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.136002    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.237919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.237883    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.238183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.238165    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.238455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.238419    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8xp\" (UniqueName: \"kubernetes.io/projected/fa4d1827-68ff-4e1b-951a-1d37bc62c062-kube-api-access-vd8xp\") pod \"test-curl-6908-5e2e20\" (UID: \"fa4d1827-68ff-4e1b-951a-1d37bc62c062\") " pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:12.238590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.238495    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.238664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.238615    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72b92\" (UniqueName: \"kubernetes.io/projected/e52e2b6e-617f-42d1-9370-c3c293fd14f4-kube-api-access-72b92\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.239383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.238673    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.239383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.238710    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84x77\" (UniqueName: \"kubernetes.io/projected/69c72c5c-5abe-4297-aa72-8679b1c3d042-kube-api-access-84x77\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.240075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.240049    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.241634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.241609    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.311955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.311864    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8xp\" (UniqueName: \"kubernetes.io/projected/fa4d1827-68ff-4e1b-951a-1d37bc62c062-kube-api-access-vd8xp\") pod \"test-curl-6908-5e2e20\" (UID: \"fa4d1827-68ff-4e1b-951a-1d37bc62c062\") " pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:12.312115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.312093    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84x77\" (UniqueName: \"kubernetes.io/projected/69c72c5c-5abe-4297-aa72-8679b1c3d042-kube-api-access-84x77\") pod \"maas-api-revoke-sub-fddba599-8bddx\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.334626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.334588    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:12.356491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.356447    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:12.367526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.367490    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.367801 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.367770    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72b92\" (UniqueName: \"kubernetes.io/projected/e52e2b6e-617f-42d1-9370-c3c293fd14f4-kube-api-access-72b92\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.367897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.367847    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.368875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.368747    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.371912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.371883    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.432273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.431957    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:12.442292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.442252    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:12.461164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.461120    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72b92\" (UniqueName: \"kubernetes.io/projected/e52e2b6e-617f-42d1-9370-c3c293fd14f4-kube-api-access-72b92\") pod \"maas-api-revoke-sub-fb6f1cdd-xckjn\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.465833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.465797    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:12.483971 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.476981    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"]
Oct 30 22:11:12.531562 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:12.531525    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c2273c_603f_402f_bbb9_fbe262767807.slice/crio-274fc663ce33f72d0d402cf0763c3525ae3c858e7d80000571a7b5963a0c4474 WatchSource:0}: Error finding container 274fc663ce33f72d0d402cf0763c3525ae3c858e7d80000571a7b5963a0c4474: Status 404 returned error can't find the container with id 274fc663ce33f72d0d402cf0763c3525ae3c858e7d80000571a7b5963a0c4474
Oct 30 22:11:12.832267 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:12.831521    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa1d9c60_0788_46d7_8ef5_d833caf75a6b.slice/crio-576f6aa91022227260275c27816db46e704333b6a5a167b0a42742558e67980b WatchSource:0}: Error finding container 576f6aa91022227260275c27816db46e704333b6a5a167b0a42742558e67980b: Status 404 returned error can't find the container with id 576f6aa91022227260275c27816db46e704333b6a5a167b0a42742558e67980b
Oct 30 22:11:12.832267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.832113    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6923-e49089"]
Oct 30 22:11:12.834138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.834056    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:11:12.952682 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:12.952651    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c72c5c_5abe_4297_aa72_8679b1c3d042.slice/crio-b55a9ebfebc8e1f43868f4c74e63551b32b52fc3c13135cfb830b3c99754d8be WatchSource:0}: Error finding container b55a9ebfebc8e1f43868f4c74e63551b32b52fc3c13135cfb830b3c99754d8be: Status 404 returned error can't find the container with id b55a9ebfebc8e1f43868f4c74e63551b32b52fc3c13135cfb830b3c99754d8be
Oct 30 22:11:12.953374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:12.953351    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"]
Oct 30 22:11:13.004997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:13.003668    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv" event={"ID":"a7c2273c-603f-402f-bbb9-fbe262767807","Type":"ContainerStarted","Data":"5054518d1befba2cacc0a7c7d74422d233c5c583d9f738b3c64db0381e6852c9"}
Oct 30 22:11:13.004997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:13.003784    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv" event={"ID":"a7c2273c-603f-402f-bbb9-fbe262767807","Type":"ContainerStarted","Data":"274fc663ce33f72d0d402cf0763c3525ae3c858e7d80000571a7b5963a0c4474"}
Oct 30 22:11:13.008313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:13.007239    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6923-e49089" event={"ID":"fa1d9c60-0788-46d7-8ef5-d833caf75a6b","Type":"ContainerStarted","Data":"576f6aa91022227260275c27816db46e704333b6a5a167b0a42742558e67980b"}
Oct 30 22:11:13.013956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:13.013920    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx" event={"ID":"69c72c5c-5abe-4297-aa72-8679b1c3d042","Type":"ContainerStarted","Data":"b55a9ebfebc8e1f43868f4c74e63551b32b52fc3c13135cfb830b3c99754d8be"}
Oct 30 22:11:13.032729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:13.032698    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-5e2e20"]
Oct 30 22:11:13.052588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:13.052436    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"]
Oct 30 22:11:13.939181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:13.939125    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv" podStartSLOduration=5.939110816 podStartE2EDuration="5.939110816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:13.936809833 +0000 UTC m=+1285.460952600" watchObservedRunningTime="2026-10-30 22:11:13.939110816 +0000 UTC m=+1285.463253570"
Oct 30 22:11:14.031409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.031371    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx" event={"ID":"69c72c5c-5abe-4297-aa72-8679b1c3d042","Type":"ContainerStarted","Data":"2b4b6fdac4db7b7571a5af4c2fcbf2e10c8375e181c78a62ee050a663c746289"}
Oct 30 22:11:14.033048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.033020    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-5e2e20" event={"ID":"fa4d1827-68ff-4e1b-951a-1d37bc62c062","Type":"ContainerStarted","Data":"df39565eca8cbb3421ab1911dbf46ef5db159897f2c77a4de9bd87b7453bc4c1"}
Oct 30 22:11:14.035254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.035180    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn" event={"ID":"e52e2b6e-617f-42d1-9370-c3c293fd14f4","Type":"ContainerStarted","Data":"bc9e4eac82daff1894d1e4e297e2e758d64e8e313a88a568b805327fe32ab645"}
Oct 30 22:11:14.035377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.035284    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn" event={"ID":"e52e2b6e-617f-42d1-9370-c3c293fd14f4","Type":"ContainerStarted","Data":"aad3fe5e55f826033e8855515d5601d9004b4688855e159395f910ee4c033cc8"}
Oct 30 22:11:14.037787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.037758    5031 generic.go:356] "Generic (PLEG): container finished" podID="a7c2273c-603f-402f-bbb9-fbe262767807" containerID="5054518d1befba2cacc0a7c7d74422d233c5c583d9f738b3c64db0381e6852c9" exitCode=0
Oct 30 22:11:14.037921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.037843    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv" event={"ID":"a7c2273c-603f-402f-bbb9-fbe262767807","Type":"ContainerDied","Data":"5054518d1befba2cacc0a7c7d74422d233c5c583d9f738b3c64db0381e6852c9"}
Oct 30 22:11:14.190910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.190797    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx" podStartSLOduration=4.190781525 podStartE2EDuration="4.190781525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:14.188821035 +0000 UTC m=+1285.712963820" watchObservedRunningTime="2026-10-30 22:11:14.190781525 +0000 UTC m=+1285.714924289"
Oct 30 22:11:14.437160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:14.437088    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn" podStartSLOduration=3.437068844 podStartE2EDuration="3.437068844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:14.428678196 +0000 UTC m=+1285.952820988" watchObservedRunningTime="2026-10-30 22:11:14.437068844 +0000 UTC m=+1285.961211608"
Oct 30 22:11:15.055617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.055576    5031 generic.go:356] "Generic (PLEG): container finished" podID="69c72c5c-5abe-4297-aa72-8679b1c3d042" containerID="2b4b6fdac4db7b7571a5af4c2fcbf2e10c8375e181c78a62ee050a663c746289" exitCode=0
Oct 30 22:11:15.056087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.055660    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx" event={"ID":"69c72c5c-5abe-4297-aa72-8679b1c3d042","Type":"ContainerDied","Data":"2b4b6fdac4db7b7571a5af4c2fcbf2e10c8375e181c78a62ee050a663c746289"}
Oct 30 22:11:15.058135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.057820    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-5e2e20" event={"ID":"fa4d1827-68ff-4e1b-951a-1d37bc62c062","Type":"ContainerStarted","Data":"4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176"}
Oct 30 22:11:15.059665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.059641    5031 generic.go:356] "Generic (PLEG): container finished" podID="e52e2b6e-617f-42d1-9370-c3c293fd14f4" containerID="bc9e4eac82daff1894d1e4e297e2e758d64e8e313a88a568b805327fe32ab645" exitCode=0
Oct 30 22:11:15.059809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.059717    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn" event={"ID":"e52e2b6e-617f-42d1-9370-c3c293fd14f4","Type":"ContainerDied","Data":"bc9e4eac82daff1894d1e4e297e2e758d64e8e313a88a568b805327fe32ab645"}
Oct 30 22:11:15.061489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.061467    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6923-e49089" event={"ID":"fa1d9c60-0788-46d7-8ef5-d833caf75a6b","Type":"ContainerStarted","Data":"8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8"}
Oct 30 22:11:15.242617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.242588    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:15.296480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.296421    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6923-e49089" podStartSLOduration=6.174543723 podStartE2EDuration="7.296406693s" totalImagesPullingTime="1.12186297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:15.293538249 +0000 UTC m=+1286.817681039" watchObservedRunningTime="2026-10-30 22:11:15.296406693 +0000 UTC m=+1286.820549457"
Oct 30 22:11:15.355065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.355030    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-xks-ca\") pod \"a7c2273c-603f-402f-bbb9-fbe262767807\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") "
Oct 30 22:11:15.355268 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.355113    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-openshift-ca\") pod \"a7c2273c-603f-402f-bbb9-fbe262767807\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") "
Oct 30 22:11:15.355268 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.355237    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c2273c-603f-402f-bbb9-fbe262767807-kube-api-access-wzfvc\" (UniqueName: \"kubernetes.io/projected/a7c2273c-603f-402f-bbb9-fbe262767807-kube-api-access-wzfvc\") pod \"a7c2273c-603f-402f-bbb9-fbe262767807\" (UID: \"a7c2273c-603f-402f-bbb9-fbe262767807\") "
Oct 30 22:11:15.355522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.355492    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-openshift-ca" pod "a7c2273c-603f-402f-bbb9-fbe262767807" (UID: "a7c2273c-603f-402f-bbb9-fbe262767807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:15.356001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.355973    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:15.357521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.357495    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-xks-ca" pod "a7c2273c-603f-402f-bbb9-fbe262767807" (UID: "a7c2273c-603f-402f-bbb9-fbe262767807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:15.357739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.357715    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c2273c-603f-402f-bbb9-fbe262767807-kube-api-access-wzfvc" pod "a7c2273c-603f-402f-bbb9-fbe262767807" (UID: "a7c2273c-603f-402f-bbb9-fbe262767807"). InnerVolumeSpecName "kube-api-access-wzfvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:15.457331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.457290    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfvc\" (UniqueName: \"kubernetes.io/projected/a7c2273c-603f-402f-bbb9-fbe262767807-kube-api-access-wzfvc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:15.457331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.457329    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c2273c-603f-402f-bbb9-fbe262767807-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:15.543790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:15.543727    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6908-5e2e20" podStartSLOduration=6.309138046 podStartE2EDuration="7.543710407s" totalImagesPullingTime="1.234572361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:15.541387647 +0000 UTC m=+1287.065530413" watchObservedRunningTime="2026-10-30 22:11:15.543710407 +0000 UTC m=+1287.067853168"
Oct 30 22:11:16.073999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.073963    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv" event={"ID":"a7c2273c-603f-402f-bbb9-fbe262767807","Type":"ContainerDied","Data":"274fc663ce33f72d0d402cf0763c3525ae3c858e7d80000571a7b5963a0c4474"}
Oct 30 22:11:16.074517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.074015    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="274fc663ce33f72d0d402cf0763c3525ae3c858e7d80000571a7b5963a0c4474"
Oct 30 22:11:16.074517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.074018    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"
Oct 30 22:11:16.308156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.308125    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:16.311635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.311607    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:16.474018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.473984    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-xks-ca\") pod \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") "
Oct 30 22:11:16.474275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.474057    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-openshift-ca\") pod \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") "
Oct 30 22:11:16.474275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.474130    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c72c5c-5abe-4297-aa72-8679b1c3d042-kube-api-access-84x77\" (UniqueName: \"kubernetes.io/projected/69c72c5c-5abe-4297-aa72-8679b1c3d042-kube-api-access-84x77\") pod \"69c72c5c-5abe-4297-aa72-8679b1c3d042\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") "
Oct 30 22:11:16.474275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.474239    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-openshift-ca\") pod \"69c72c5c-5abe-4297-aa72-8679b1c3d042\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") "
Oct 30 22:11:16.474502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.474298    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52e2b6e-617f-42d1-9370-c3c293fd14f4-kube-api-access-72b92\" (UniqueName: \"kubernetes.io/projected/e52e2b6e-617f-42d1-9370-c3c293fd14f4-kube-api-access-72b92\") pod \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\" (UID: \"e52e2b6e-617f-42d1-9370-c3c293fd14f4\") "
Oct 30 22:11:16.474502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.474461    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-xks-ca\") pod \"69c72c5c-5abe-4297-aa72-8679b1c3d042\" (UID: \"69c72c5c-5abe-4297-aa72-8679b1c3d042\") "
Oct 30 22:11:16.474623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.474535    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-openshift-ca" pod "e52e2b6e-617f-42d1-9370-c3c293fd14f4" (UID: "e52e2b6e-617f-42d1-9370-c3c293fd14f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:16.474931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.474901    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-openshift-ca" pod "69c72c5c-5abe-4297-aa72-8679b1c3d042" (UID: "69c72c5c-5abe-4297-aa72-8679b1c3d042"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:16.475356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.475296    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:16.475356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.475328    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:16.481139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.478524    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-xks-ca" pod "69c72c5c-5abe-4297-aa72-8679b1c3d042" (UID: "69c72c5c-5abe-4297-aa72-8679b1c3d042"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:16.481139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.478776    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-xks-ca" pod "e52e2b6e-617f-42d1-9370-c3c293fd14f4" (UID: "e52e2b6e-617f-42d1-9370-c3c293fd14f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:16.481139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.480806    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52e2b6e-617f-42d1-9370-c3c293fd14f4-kube-api-access-72b92" pod "e52e2b6e-617f-42d1-9370-c3c293fd14f4" (UID: "e52e2b6e-617f-42d1-9370-c3c293fd14f4"). InnerVolumeSpecName "kube-api-access-72b92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:16.484017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.483985    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c72c5c-5abe-4297-aa72-8679b1c3d042-kube-api-access-84x77" pod "69c72c5c-5abe-4297-aa72-8679b1c3d042" (UID: "69c72c5c-5abe-4297-aa72-8679b1c3d042"). InnerVolumeSpecName "kube-api-access-84x77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:16.577468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.577431    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84x77\" (UniqueName: \"kubernetes.io/projected/69c72c5c-5abe-4297-aa72-8679b1c3d042-kube-api-access-84x77\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:16.577468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.577471    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72b92\" (UniqueName: \"kubernetes.io/projected/e52e2b6e-617f-42d1-9370-c3c293fd14f4-kube-api-access-72b92\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:16.577679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.577489    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c72c5c-5abe-4297-aa72-8679b1c3d042-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:16.577679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.577503    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e2b6e-617f-42d1-9370-c3c293fd14f4-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:16.929530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.929492    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6923-e49089"]
Oct 30 22:11:16.938379 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.938341    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6923-e49089"]
Oct 30 22:11:16.965041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.965004    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6908-5e2e20"]
Oct 30 22:11:16.972541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:16.972483    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6908-5e2e20"]
Oct 30 22:11:17.086084 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.086059    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"
Oct 30 22:11:17.086084 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.086070    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx" event={"ID":"69c72c5c-5abe-4297-aa72-8679b1c3d042","Type":"ContainerDied","Data":"b55a9ebfebc8e1f43868f4c74e63551b32b52fc3c13135cfb830b3c99754d8be"}
Oct 30 22:11:17.086592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.086148    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b55a9ebfebc8e1f43868f4c74e63551b32b52fc3c13135cfb830b3c99754d8be"
Oct 30 22:11:17.088032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.088005    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"
Oct 30 22:11:17.088199 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.088023    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn" event={"ID":"e52e2b6e-617f-42d1-9370-c3c293fd14f4","Type":"ContainerDied","Data":"aad3fe5e55f826033e8855515d5601d9004b4688855e159395f910ee4c033cc8"}
Oct 30 22:11:17.088361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.088236    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad3fe5e55f826033e8855515d5601d9004b4688855e159395f910ee4c033cc8"
Oct 30 22:11:17.088361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.088287    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6908-5e2e20" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" containerName="test-curl-6908-5e2e20" containerID="cri-o://4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176" gracePeriod=2
Oct 30 22:11:17.088478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.088380    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6923-e49089" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" containerName="test-curl-6923-e49089" containerID="cri-o://8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8" gracePeriod=2
Oct 30 22:11:17.141262 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:17.141194    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6908-5e2e20\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:17.193130 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:17.193022    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6923-e49089\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:17.356372 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:17.356335    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6908-5e2e20\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:17.385932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.385880    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:17.408403 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:17.408363    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6923-e49089\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:17.758155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.758123    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:17.976641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:17.976594    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:18.596039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:18.596003    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:18.696752 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:18.696708    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6923-e49089\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:18.748677 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:18.748636    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6908-5e2e20\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:19.396690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.396661    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:19.401140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.401115    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:19.422309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.422278    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4d1827-68ff-4e1b-951a-1d37bc62c062-kube-api-access-vd8xp\" (UniqueName: \"kubernetes.io/projected/fa4d1827-68ff-4e1b-951a-1d37bc62c062-kube-api-access-vd8xp\") pod \"fa4d1827-68ff-4e1b-951a-1d37bc62c062\" (UID: \"fa4d1827-68ff-4e1b-951a-1d37bc62c062\") "
Oct 30 22:11:19.422490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.422477    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1d9c60-0788-46d7-8ef5-d833caf75a6b-kube-api-access-n9wbc\" (UniqueName: \"kubernetes.io/projected/fa1d9c60-0788-46d7-8ef5-d833caf75a6b-kube-api-access-n9wbc\") pod \"fa1d9c60-0788-46d7-8ef5-d833caf75a6b\" (UID: \"fa1d9c60-0788-46d7-8ef5-d833caf75a6b\") "
Oct 30 22:11:19.424691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.424664    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4d1827-68ff-4e1b-951a-1d37bc62c062-kube-api-access-vd8xp" pod "fa4d1827-68ff-4e1b-951a-1d37bc62c062" (UID: "fa4d1827-68ff-4e1b-951a-1d37bc62c062"). InnerVolumeSpecName "kube-api-access-vd8xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:19.424981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.424960    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1d9c60-0788-46d7-8ef5-d833caf75a6b-kube-api-access-n9wbc" pod "fa1d9c60-0788-46d7-8ef5-d833caf75a6b" (UID: "fa1d9c60-0788-46d7-8ef5-d833caf75a6b"). InnerVolumeSpecName "kube-api-access-n9wbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:19.524648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.524615    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9wbc\" (UniqueName: \"kubernetes.io/projected/fa1d9c60-0788-46d7-8ef5-d833caf75a6b-kube-api-access-n9wbc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:19.524648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.524644    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8xp\" (UniqueName: \"kubernetes.io/projected/fa4d1827-68ff-4e1b-951a-1d37bc62c062-kube-api-access-vd8xp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:19.946049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:19.945993    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:20.121946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.121903    5031 generic.go:356] "Generic (PLEG): container finished" podID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" containerID="4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176" exitCode=137
Oct 30 22:11:20.122111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.121967    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:20.122111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.122005    5031 scope.go:122] "RemoveContainer" containerID="4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176"
Oct 30 22:11:20.124170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.124139    5031 generic.go:356] "Generic (PLEG): container finished" podID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" containerID="8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8" exitCode=137
Oct 30 22:11:20.124309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.124194    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:20.135416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.135392    5031 scope.go:122] "RemoveContainer" containerID="4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176"
Oct 30 22:11:20.135736 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:20.135716    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176\": container with ID starting with 4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176 not found: ID does not exist" containerID="4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176"
Oct 30 22:11:20.135784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.135746    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176"} err="failed to get container status \"4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176\": rpc error: code = NotFound desc = could not find container \"4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176\": container with ID starting with 4103fdc6c061594f86a1f1161ea8e0e2a8460d5bf1466ef3a4aaeed3b8a54176 not found: ID does not exist"
Oct 30 22:11:20.135784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.135767    5031 scope.go:122] "RemoveContainer" containerID="8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8"
Oct 30 22:11:20.147044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.147024    5031 scope.go:122] "RemoveContainer" containerID="8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8"
Oct 30 22:11:20.147367 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:20.147345    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8\": container with ID starting with 8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8 not found: ID does not exist" containerID="8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8"
Oct 30 22:11:20.147421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.147377    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8"} err="failed to get container status \"8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8\": rpc error: code = NotFound desc = could not find container \"8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8\": container with ID starting with 8873e9fbd423cc25e41cea35f5bc293b7b0533d9be9b6fc9f9ba106c18b9f3f8 not found: ID does not exist"
Oct 30 22:11:20.167064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.167036    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:20.185914 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:20.185869    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6908-5e2e20\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:20.237863 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:20.237767    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6923-e49089\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:20.289562 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:20.289518    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6908-5e2e20\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" pod="opendatahub/test-curl-6908-5e2e20"
Oct 30 22:11:20.341191 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:20.341150    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6923-e49089\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" pod="opendatahub/test-curl-6923-e49089"
Oct 30 22:11:20.415128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.415087    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:20.544697 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:20.544605    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:11:20.655645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.655598    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" path="/var/lib/kubelet/pods/fa1d9c60-0788-46d7-8ef5-d833caf75a6b/volumes"
Oct 30 22:11:20.656292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.656267    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" path="/var/lib/kubelet/pods/fa4d1827-68ff-4e1b-951a-1d37bc62c062/volumes"
Oct 30 22:11:20.791569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.791408    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"]
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792695    5031 cpu_manager.go:414] "containerMap: removing container" podUID="69c72c5c-5abe-4297-aa72-8679b1c3d042" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792719    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c72c5c-5abe-4297-aa72-8679b1c3d042" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792819    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e52e2b6e-617f-42d1-9370-c3c293fd14f4" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792827    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52e2b6e-617f-42d1-9370-c3c293fd14f4" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792860    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a7c2273c-603f-402f-bbb9-fbe262767807" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792869    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c2273c-603f-402f-bbb9-fbe262767807" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792885    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" containerName="test-curl-6923-e49089"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792892    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" containerName="test-curl-6923-e49089"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792903    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" containerName="test-curl-6908-5e2e20"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.792911    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" containerName="test-curl-6908-5e2e20"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.793085    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c2273c-603f-402f-bbb9-fbe262767807" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.793101    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1d9c60-0788-46d7-8ef5-d833caf75a6b" containerName="test-curl-6923-e49089"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.793110    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4d1827-68ff-4e1b-951a-1d37bc62c062" containerName="test-curl-6908-5e2e20"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.793122    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c72c5c-5abe-4297-aa72-8679b1c3d042" containerName="revoke-keys"
Oct 30 22:11:20.793195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.793140    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52e2b6e-617f-42d1-9370-c3c293fd14f4" containerName="revoke-keys"
Oct 30 22:11:20.807322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.804588    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850282    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-envoy\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850400    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-socket\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850429    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-credential-socket\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850506    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-certs\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850575    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdrx\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-kube-api-access-dmdrx\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850689    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-data\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850728    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aabc72e-c473-4680-9422-c32ddd053f82-istio-podinfo\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850799    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aabc72e-c473-4680-9422-c32ddd053f82-istiod-ca-cert\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.896317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.850825    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-istio-token\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:20.923324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:20.897490    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4cce86f1-gw-openshift-default-dockercfg-xmqnp\""
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.041786    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.062457    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdrx\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-kube-api-access-dmdrx\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.062618    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-data\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.062657    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aabc72e-c473-4680-9422-c32ddd053f82-istio-podinfo\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.062734    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aabc72e-c473-4680-9422-c32ddd053f82-istiod-ca-cert\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.062759    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-istio-token\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.062897    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-envoy\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.063007    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-socket\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.063029    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-credential-socket\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.063157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-certs\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.064570    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aabc72e-c473-4680-9422-c32ddd053f82-istiod-ca-cert\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.065592    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-data\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.065797    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-socket\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.066248    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-credential-socket\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.067726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.066497    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-certs\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.078047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.077765    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-envoy\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.084073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.084043    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aabc72e-c473-4680-9422-c32ddd053f82-istio-podinfo\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.124869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.124778    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"]
Oct 30 22:11:21.146465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.145077    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-istio-token\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.147402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.147369    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdrx\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-kube-api-access-dmdrx\") pod \"e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") " pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.150182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.150154    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:21.574928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.574885    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:21.675844 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:21.675814    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aabc72e_c473_4680_9422_c32ddd053f82.slice/crio-825fe196af3afde9d5119661b1b421d155ba8f691c93a04b2bfdfd135a9047a0 WatchSource:0}: Error finding container 825fe196af3afde9d5119661b1b421d155ba8f691c93a04b2bfdfd135a9047a0: Status 404 returned error can't find the container with id 825fe196af3afde9d5119661b1b421d155ba8f691c93a04b2bfdfd135a9047a0
Oct 30 22:11:21.677312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.677280    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"]
Oct 30 22:11:21.679167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.679128    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:21.679340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.679259    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:21.679340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:21.679305    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:22.168306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:22.168265    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" event={"ID":"9aabc72e-c473-4680-9422-c32ddd053f82","Type":"ContainerStarted","Data":"3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4"}
Oct 30 22:11:22.168800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:22.168345    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" event={"ID":"9aabc72e-c473-4680-9422-c32ddd053f82","Type":"ContainerStarted","Data":"825fe196af3afde9d5119661b1b421d155ba8f691c93a04b2bfdfd135a9047a0"}
Oct 30 22:11:22.300408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:22.300343    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podStartSLOduration=2.300323703 podStartE2EDuration="2.300323703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:22.296863914 +0000 UTC m=+1293.821006706" watchObservedRunningTime="2026-10-30 22:11:22.300323703 +0000 UTC m=+1293.824466466"
Oct 30 22:11:22.625428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:22.625388    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:23.150384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:23.150349    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:23.157309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:23.157283    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:23.178521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:23.178493    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:23.179713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:23.179688    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:11:23.317182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:23.317152    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:24.209961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.209916    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"]
Oct 30 22:11:24.216465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.216446    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.273272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.271625    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-08235cc9-openshift-default-dockercfg-dwrdg\""
Oct 30 22:11:24.334595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.334560    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"]
Oct 30 22:11:24.344779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.344744    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-envoy\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.344976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.344894    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-certs\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.344976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.344947    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-socket\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.345324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.345295    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-podinfo\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.345430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.345390    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a805d-46e7-499f-b443-fde8acc5863a-istiod-ca-cert\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.345524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.345507    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-data\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.345581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.345547    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-credential-socket\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.345714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.345692    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-token\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.345787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.345768    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjpf\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-kube-api-access-jjjpf\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.383004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.382967    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:24.447605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.447570    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-podinfo\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.447816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.447633    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a805d-46e7-499f-b443-fde8acc5863a-istiod-ca-cert\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.447816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.447743    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-data\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.447816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.447797    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-credential-socket\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.447993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.447886    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-token\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.447993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.447952    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjpf\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-kube-api-access-jjjpf\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448023    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-envoy\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448095    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-certs\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448146    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-socket\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448163    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-data\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448269    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-credential-socket\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448573    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-certs\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448607    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a805d-46e7-499f-b443-fde8acc5863a-istiod-ca-cert\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.448648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.448612    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-socket\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.450789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.450764    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-podinfo\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.451020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.451000    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-envoy\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.516507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.516433    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-token\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.516507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.516484    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjpf\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-kube-api-access-jjjpf\") pod \"e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") " pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.538761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.538725    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:24.810189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:24.808930    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:25.038632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:25.038598    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:25.202380 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:25.202263    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6a805d_46e7_499f_b443_fde8acc5863a.slice/crio-1866cccbcb32bb5d355031aa1b96b6abd7f9dc53953076c49484bb27ee6f59ff WatchSource:0}: Error finding container 1866cccbcb32bb5d355031aa1b96b6abd7f9dc53953076c49484bb27ee6f59ff: Status 404 returned error can't find the container with id 1866cccbcb32bb5d355031aa1b96b6abd7f9dc53953076c49484bb27ee6f59ff
Oct 30 22:11:25.209459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:25.209423    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"]
Oct 30 22:11:25.217504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:25.217448    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:25.217855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:25.217535    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:25.217855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:25.217570    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:25.276729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:25.276673    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:25.498112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:25.498021    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:26.221951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:26.221913    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" event={"ID":"1f6a805d-46e7-499f-b443-fde8acc5863a","Type":"ContainerStarted","Data":"df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325"}
Oct 30 22:11:26.222456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:26.221976    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" event={"ID":"1f6a805d-46e7-499f-b443-fde8acc5863a","Type":"ContainerStarted","Data":"1866cccbcb32bb5d355031aa1b96b6abd7f9dc53953076c49484bb27ee6f59ff"}
Oct 30 22:11:26.405292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:26.405237    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" podStartSLOduration=2.405220584 podStartE2EDuration="2.405220584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:26.398743249 +0000 UTC m=+1297.922886032" watchObservedRunningTime="2026-10-30 22:11:26.405220584 +0000 UTC m=+1297.929363343"
Oct 30 22:11:26.562345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:26.562197    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:26.568338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:26.568309    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:26.907618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:26.907582    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:27.135276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:27.135201    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:27.236733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:27.236638    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:27.237907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:27.237885    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:11:27.358247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:27.358154    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:28.237780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:28.237731    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:28.495367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:28.493163    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:28.736820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:28.736772    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:29.401909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.401873    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:29.626521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.626483    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:29.752266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.752155    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"]
Oct 30 22:11:29.758167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.758142    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:29.816249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.816195    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"]
Oct 30 22:11:29.834828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.834793    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:29.835062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.834909    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8mh\" (UniqueName: \"kubernetes.io/projected/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-kube-api-access-pw8mh\") pod \"payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:29.862241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.859388    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4cce86f1\""
Oct 30 22:11:29.862241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.859571    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4cce86f1-dockercfg-lxkcl\""
Oct 30 22:11:29.962257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.945375    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:29.962257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.945471    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8mh\" (UniqueName: \"kubernetes.io/projected/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-kube-api-access-pw8mh\") pod \"payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:29.967972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.967881    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:30.006440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:29.995769    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:30.043266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.026271    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"]
Oct 30 22:11:30.062969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.059852    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.139083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.139043    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"]
Oct 30 22:11:30.142053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.141934    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8mh\" (UniqueName: \"kubernetes.io/projected/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-kube-api-access-pw8mh\") pod \"payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:30.168808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.168773    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49twr\" (UniqueName: \"kubernetes.io/projected/7ab19abc-fc51-48ca-8f50-f65f37837d6b-kube-api-access-49twr\") pod \"payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.169554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.169528    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab19abc-fc51-48ca-8f50-f65f37837d6b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.226239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.226186    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:30.271005 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.270913    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49twr\" (UniqueName: \"kubernetes.io/projected/7ab19abc-fc51-48ca-8f50-f65f37837d6b-kube-api-access-49twr\") pod \"payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.271191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.271114    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab19abc-fc51-48ca-8f50-f65f37837d6b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.271961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.271933    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab19abc-fc51-48ca-8f50-f65f37837d6b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.335641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.335608    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49twr\" (UniqueName: \"kubernetes.io/projected/7ab19abc-fc51-48ca-8f50-f65f37837d6b-kube-api-access-49twr\") pod \"payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.374345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.374308    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:30.500986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.499534    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:30.796003 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:30.795965    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b930a7_116d_4fd2_ae48_d9d77b1ad933.slice/crio-5867fe5d78d0ff26fc94edc97896b339768182b6e45efb63ce86ddc75e1892ea WatchSource:0}: Error finding container 5867fe5d78d0ff26fc94edc97896b339768182b6e45efb63ce86ddc75e1892ea: Status 404 returned error can't find the container with id 5867fe5d78d0ff26fc94edc97896b339768182b6e45efb63ce86ddc75e1892ea
Oct 30 22:11:30.797003 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.796974    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"]
Oct 30 22:11:30.919549 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:30.919511    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab19abc_fc51_48ca_8f50_f65f37837d6b.slice/crio-0c31d5d41199ac048ba94c23955c6919175156cebe2ca200e94fcf88671cb63d WatchSource:0}: Error finding container 0c31d5d41199ac048ba94c23955c6919175156cebe2ca200e94fcf88671cb63d: Status 404 returned error can't find the container with id 0c31d5d41199ac048ba94c23955c6919175156cebe2ca200e94fcf88671cb63d
Oct 30 22:11:30.925673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:30.925527    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"]
Oct 30 22:11:31.289369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:31.289331    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf" event={"ID":"f3b930a7-116d-4fd2-ae48-d9d77b1ad933","Type":"ContainerStarted","Data":"7d7b8ae03a83f3158e01445295e48e32b77fb88ba95c48ae4d243705791b218b"}
Oct 30 22:11:31.289561 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:31.289396    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf" event={"ID":"f3b930a7-116d-4fd2-ae48-d9d77b1ad933","Type":"ContainerStarted","Data":"5867fe5d78d0ff26fc94edc97896b339768182b6e45efb63ce86ddc75e1892ea"}
Oct 30 22:11:31.289561 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:31.289429    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:31.291515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:31.291484    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245" event={"ID":"7ab19abc-fc51-48ca-8f50-f65f37837d6b","Type":"ContainerStarted","Data":"6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c"}
Oct 30 22:11:31.291689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:31.291535    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:31.291689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:31.291552    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245" event={"ID":"7ab19abc-fc51-48ca-8f50-f65f37837d6b","Type":"ContainerStarted","Data":"0c31d5d41199ac048ba94c23955c6919175156cebe2ca200e94fcf88671cb63d"}
Oct 30 22:11:31.435422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:31.435364    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf" podStartSLOduration=2.435347642 podStartE2EDuration="2.435347642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:31.408853663 +0000 UTC m=+1302.932996451" watchObservedRunningTime="2026-10-30 22:11:31.435347642 +0000 UTC m=+1302.959490444"
Oct 30 22:11:32.376193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:32.376161    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:32.593175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:32.593133    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:32.829655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:32.829552    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:33.063355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:33.063322    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:34.081597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.081563    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:34.370294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.370250    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:34.817961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.817855    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245" podStartSLOduration=5.817839152 podStartE2EDuration="5.817839152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:31.539484259 +0000 UTC m=+1303.063627035" watchObservedRunningTime="2026-10-30 22:11:34.817839152 +0000 UTC m=+1306.341981916"
Oct 30 22:11:34.820055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.820026    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"]
Oct 30 22:11:34.824993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.824975    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.939518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.939477    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"]
Oct 30 22:11:34.966755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.966718    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.966949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.966887    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728354e7-a85a-46da-9c55-e30df62bc50c-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.967021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.967005    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.967135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.967118    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.967245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.967226    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhcv\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-kube-api-access-4dhcv\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.967398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.967382    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728354e7-a85a-46da-9c55-e30df62bc50c-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.967517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.967424    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.967517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.967504    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:34.967596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:34.967542    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.027496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.027460    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.100730    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.100904    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.100986    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhcv\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-kube-api-access-4dhcv\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.101194    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728354e7-a85a-46da-9c55-e30df62bc50c-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.101268    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.101311    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.101340    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.101453    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.101644    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728354e7-a85a-46da-9c55-e30df62bc50c-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.103248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.102755    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728354e7-a85a-46da-9c55-e30df62bc50c-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.104279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.103387    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.110158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.107567    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.110158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.108153    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.110158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.108431    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.137730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.137688    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.183920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.183882    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728354e7-a85a-46da-9c55-e30df62bc50c-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.194106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.188516    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhcv\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-kube-api-access-4dhcv\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.214244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.213178    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:35.214244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.213511    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:11:35.256364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.256332    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:35.445330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:35.444899    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:36.242814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:36.242768    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"]
Oct 30 22:11:36.248432 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:36.248398    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728354e7_a85a_46da_9c55_e30df62bc50c.slice/crio-6dae8a190de9feeee835fd7ac9ef332d5fce6e51d465e99b4cc186a2c1a1e163 WatchSource:0}: Error finding container 6dae8a190de9feeee835fd7ac9ef332d5fce6e51d465e99b4cc186a2c1a1e163: Status 404 returned error can't find the container with id 6dae8a190de9feeee835fd7ac9ef332d5fce6e51d465e99b4cc186a2c1a1e163
Oct 30 22:11:36.252405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:36.252357    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:36.252517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:36.252478    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:36.252594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:36.252531    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:36.536648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:36.534911    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" event={"ID":"728354e7-a85a-46da-9c55-e30df62bc50c","Type":"ContainerStarted","Data":"6dae8a190de9feeee835fd7ac9ef332d5fce6e51d465e99b4cc186a2c1a1e163"}
Oct 30 22:11:37.408306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.408267    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"]
Oct 30 22:11:37.450738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.450701    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.527485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.522274    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"]
Oct 30 22:11:37.552708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.552662    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.573031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.572991    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.573822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.573794    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x9s\" (UniqueName: \"kubernetes.io/projected/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-kube-api-access-89x9s\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.574288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.573625    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:11:37.574416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.573545    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:37.574525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.573646    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:37.574597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.573702    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:37.679685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.679589    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.679962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.679939    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x9s\" (UniqueName: \"kubernetes.io/projected/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-kube-api-access-89x9s\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.706330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.706291    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.737231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.715989    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.802235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.795649    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.828348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.828310    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" event={"ID":"728354e7-a85a-46da-9c55-e30df62bc50c","Type":"ContainerStarted","Data":"9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc"}
Oct 30 22:11:37.895104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.895064    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x9s\" (UniqueName: \"kubernetes.io/projected/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-kube-api-access-89x9s\") pod \"maas-api-revoke-sub-f7376483-jtczp\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:37.905101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.904316    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:11:37.974297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:37.968404    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podStartSLOduration=3.968384709 podStartE2EDuration="3.968384709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:37.967830901 +0000 UTC m=+1309.491973694" watchObservedRunningTime="2026-10-30 22:11:37.968384709 +0000 UTC m=+1309.492527475"
Oct 30 22:11:38.043242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.012437    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.117239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.114393    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:38.321391 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.321341    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.351255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.334681    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:11:38.351599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.351555    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.388364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.388299    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:11:38.492295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.477183    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:11:38.492295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.492127    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:38.595254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.570802    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.631243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.602305    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:38.631243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.602421    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:38.858082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.814474    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:38.858082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.814760    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:11:38.878243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.870686    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:39.008285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:38.904742    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.149288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.016947    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.194429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.194375    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:39.275343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.251664    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:11:39.279497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.279467    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:11:39.334423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.334381    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:39.334710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.334669    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.360287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.336289    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:11:39.387240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.361854    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:11:39.387823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.370701    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.650630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.428530    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:11:39.651251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.574312    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:39.651413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.609184    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:11:39.720702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.685749    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:11:39.720702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.685843    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:11:39.720702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.710725    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:11:39.720702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:39.710821    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:11:40.184238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.170543    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.420778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.387573    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.597128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.596441    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:40.597128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.596546    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.705181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.705069    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.769580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.766709    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.769580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.767020    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:11:40.769580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.734712    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:40.769580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.767151    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:40.821747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.731335    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:40.822793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.822739    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:40.823006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:40.822978    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.125739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.125698    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:41.126067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.126029    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.399497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.390346    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.460544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.460492    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.473259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.461942    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:41.526569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.526518    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:11:41.568707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.526826    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:11:41.640166 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.640124    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:41.653446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.653286    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.723318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.683278    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:41.898262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.751608    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.898262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.842903    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.898262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.879886    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:41.936747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.936643    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn" podUID="8a6d5c6a-ec67-4875-960c-916306762084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 22:11:41.978387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.945546    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:11:42.094884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:41.212939    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:42.481362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.481310    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:11:42.531251 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:42.531196    5031 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.735s"
Oct 30 22:11:42.531541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.531526    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:11:42.531755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.531743    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:11:42.531898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.531887    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:11:42.532004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.531993    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"]
Oct 30 22:11:42.549419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.549384    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"]
Oct 30 22:11:42.549717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.549689    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:42.551265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.511906    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:42.571236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.556918    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:42.575251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.575177    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:42.596657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.596629    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:11:42.621933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.621856    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:42.625358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.623550    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:11:42.627994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.627779    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"]
Oct 30 22:11:42.664938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.664238    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-643a82de\""
Oct 30 22:11:42.664938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.664681    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-643a82de-dockercfg-p9ztg\""
Oct 30 22:11:42.704660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.704622    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-plugins-config-volume\") pod \"payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") " pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:42.704871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.704851    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqt9\" (UniqueName: \"kubernetes.io/projected/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-kube-api-access-2cqt9\") pod \"payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") " pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:42.781303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.780727    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:42.788806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.788770    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:11:42.789459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.789428    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:11:42.790710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.790693    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:11:42.790894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.790879    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:11:42.791017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.791006    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:11:42.827167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.827039    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-plugins-config-volume\") pod \"payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") " pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:42.827657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.827613    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqt9\" (UniqueName: \"kubernetes.io/projected/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-kube-api-access-2cqt9\") pod \"payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") " pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:42.831904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.830555    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-plugins-config-volume\") pod \"payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") " pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:42.882193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.882143    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:11:42.892587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.892555    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:11:42.927426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.905297    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:11:42.927426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.905385    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:11:42.997449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:42.997401    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqt9\" (UniqueName: \"kubernetes.io/projected/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-kube-api-access-2cqt9\") pod \"payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") " pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:43.049920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:43.049645    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:11:43.088023 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:43.087977    5031 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dde23f_fad0_437d_8fb2_21ab8d01ab0d.slice/crio-fe3b30914b9b912bf0af33fb234373d4cac27007bfcf755c75e2df6f4f2fde5d\": RecentStats: unable to find data in memory cache]"
Oct 30 22:11:43.271400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:43.271370    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:43.459422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:43.458854    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:43.459422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:43.458912    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:43.667495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:43.667408    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp" event={"ID":"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d","Type":"ContainerStarted","Data":"6dda5b9423de84102e4518ea9b76f413e2a99fa7fca8878321e1128f7be0c00a"}
Oct 30 22:11:43.667495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:43.667470    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp" event={"ID":"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d","Type":"ContainerStarted","Data":"fe3b30914b9b912bf0af33fb234373d4cac27007bfcf755c75e2df6f4f2fde5d"}
Oct 30 22:11:43.738830 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:11:43.738784    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cf1e3b_0931_44f1_b3e3_cb1f7f8f948f.slice/crio-99b125c6cf99285fbb5ecdb08c0cf99ad62bf62af8f5759e9afb45579db36446 WatchSource:0}: Error finding container 99b125c6cf99285fbb5ecdb08c0cf99ad62bf62af8f5759e9afb45579db36446: Status 404 returned error can't find the container with id 99b125c6cf99285fbb5ecdb08c0cf99ad62bf62af8f5759e9afb45579db36446
Oct 30 22:11:43.741133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:43.741111    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"]
Oct 30 22:11:44.473964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:44.458092    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:44.474515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:44.474352    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:44.682310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:44.682275    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55" event={"ID":"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f","Type":"ContainerStarted","Data":"f888b77b3a04074c206da4c8db89e6921ff90da255ea7ba0c72b7c5b6e95a754"}
Oct 30 22:11:44.682484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:44.682336    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55" event={"ID":"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f","Type":"ContainerStarted","Data":"99b125c6cf99285fbb5ecdb08c0cf99ad62bf62af8f5759e9afb45579db36446"}
Oct 30 22:11:44.682484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:44.682366    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:44.684476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:44.684444    5031 generic.go:356] "Generic (PLEG): container finished" podID="d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" containerID="6dda5b9423de84102e4518ea9b76f413e2a99fa7fca8878321e1128f7be0c00a" exitCode=0
Oct 30 22:11:44.684621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:44.684526    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp" event={"ID":"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d","Type":"ContainerDied","Data":"6dda5b9423de84102e4518ea9b76f413e2a99fa7fca8878321e1128f7be0c00a"}
Oct 30 22:11:45.184765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:45.184704    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55" podStartSLOduration=6.184688952 podStartE2EDuration="6.184688952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:45.182227621 +0000 UTC m=+1316.706370380" watchObservedRunningTime="2026-10-30 22:11:45.184688952 +0000 UTC m=+1316.708831717"
Oct 30 22:11:45.447627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:45.447529    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:45.447627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:45.447597    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:45.983807 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:45.983781    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:46.094929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.094899    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-openshift-ca\") pod \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") "
Oct 30 22:11:46.095159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.095142    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-xks-ca\") pod \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") "
Oct 30 22:11:46.095297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.095268    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-kube-api-access-89x9s\" (UniqueName: \"kubernetes.io/projected/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-kube-api-access-89x9s\") pod \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\" (UID: \"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d\") "
Oct 30 22:11:46.095374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.095342    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-openshift-ca" pod "d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" (UID: "d9dde23f-fad0-437d-8fb2-21ab8d01ab0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:46.095967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.095864    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:46.097711 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.097686    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-kube-api-access-89x9s" pod "d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" (UID: "d9dde23f-fad0-437d-8fb2-21ab8d01ab0d"). InnerVolumeSpecName "kube-api-access-89x9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:46.097797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.097777    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-xks-ca" pod "d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" (UID: "d9dde23f-fad0-437d-8fb2-21ab8d01ab0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:46.174168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.174121    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:46.197698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.197653    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:46.197698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.197698    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89x9s\" (UniqueName: \"kubernetes.io/projected/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d-kube-api-access-89x9s\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:46.447680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.447586    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:46.447680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.447665    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:46.451447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.451417    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:46.748720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.748626    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"
Oct 30 22:11:46.748720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.748645    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp" event={"ID":"d9dde23f-fad0-437d-8fb2-21ab8d01ab0d","Type":"ContainerDied","Data":"fe3b30914b9b912bf0af33fb234373d4cac27007bfcf755c75e2df6f4f2fde5d"}
Oct 30 22:11:46.748720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:46.748704    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3b30914b9b912bf0af33fb234373d4cac27007bfcf755c75e2df6f4f2fde5d"
Oct 30 22:11:47.446730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:47.446691    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:11:47.447140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:47.446761    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 22:11:47.562644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:47.562607    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:48.564736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:48.564699    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:48.565311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:48.565287    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:48.566041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:48.566024    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:11:49.225045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.225013    5031 scope.go:122] "RemoveContainer" containerID="1ada363e8bdd980ce5990ea2db18a17700e3515f8c15dfb28c818f19bd258c7b"
Oct 30 22:11:49.237611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.237572    5031 scope.go:122] "RemoveContainer" containerID="74f44971c5ad96c6522295c1389fe25e1ed6f4144dffb0cfb02ae31a213a5914"
Oct 30 22:11:49.257694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.257667    5031 scope.go:122] "RemoveContainer" containerID="bb936b5790b2c2c492bccb573d3c43d15e0d796fd1e1ad3a40ae4856f5585773"
Oct 30 22:11:49.269441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.269419    5031 scope.go:122] "RemoveContainer" containerID="61735f08f852649ace0a5faa7ea7efed8033084f76ab45359caa612c2dc81daf"
Oct 30 22:11:49.280868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.280844    5031 scope.go:122] "RemoveContainer" containerID="9b13f510e650ab6bd7efd1fad647e9b9fd0795381157d8710d1a8078ffb5878b"
Oct 30 22:11:49.292988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.292967    5031 scope.go:122] "RemoveContainer" containerID="00fe5c4cb811463da1c1fceb5760a433a6a410b4a180cbda50233e8f885b60dd"
Oct 30 22:11:49.304344 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.304323    5031 scope.go:122] "RemoveContainer" containerID="53caea505561cc56d364666c95c8c08e258c18465a27c436326473e4896a2af7"
Oct 30 22:11:49.318033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.318010    5031 scope.go:122] "RemoveContainer" containerID="ceb42943c6075a7927bcb9bbd39c554219bad108267da8025dabf7674762a235"
Oct 30 22:11:49.330384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.330362    5031 scope.go:122] "RemoveContainer" containerID="e96899cd0669367ee17e5cfa356068d4c3fdf9dd6dcd46b6aa1f9f9d72220231"
Oct 30 22:11:49.342157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.342137    5031 scope.go:122] "RemoveContainer" containerID="5e9b02ab3b96e4ae78800453160dad9fbd27d0a1575b00789af8348a6a6c70b9"
Oct 30 22:11:49.365443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.365413    5031 scope.go:122] "RemoveContainer" containerID="df2c5a8a55551d03c64bae35c84cf751c6d92248f4b41360eec8340cc9690010"
Oct 30 22:11:49.377873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.377848    5031 scope.go:122] "RemoveContainer" containerID="6111937e7902d78902ef965bf8a339ff0a888a83ce3b390e358518131ea55c47"
Oct 30 22:11:49.743536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:49.743452    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:50.546167 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:11:50.546123    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:11:52.701490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:52.701460    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:55.740190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:55.740156    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:11:57.825270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:57.825179    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.054701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:11:58.054660    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:00.191998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.191961    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"]
Oct 30 22:12:00.192997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.192972    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" containerName="revoke-keys"
Oct 30 22:12:00.192997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.192998    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" containerName="revoke-keys"
Oct 30 22:12:00.193259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.193193    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" containerName="revoke-keys"
Oct 30 22:12:00.201033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.201005    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.265888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.265832    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"]
Oct 30 22:12:00.282750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.282714    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.282959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.282838    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.283033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.283013    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4d6q\" (UniqueName: \"kubernetes.io/projected/c25b4244-cc29-41ec-b566-19242a82d8bd-kube-api-access-n4d6q\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.300456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.300424    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:00.300707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.300441    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:00.300834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.300451    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:12:00.306446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.306421    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:00.385826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.385779    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4d6q\" (UniqueName: \"kubernetes.io/projected/c25b4244-cc29-41ec-b566-19242a82d8bd-kube-api-access-n4d6q\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.386056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.385984    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.386127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.386076    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.386785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.386753    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.389454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.389410    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.456085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.456000    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4d6q\" (UniqueName: \"kubernetes.io/projected/c25b4244-cc29-41ec-b566-19242a82d8bd-kube-api-access-n4d6q\") pod \"maas-api-revoke-sub-5d9d69f7-5h5rg\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.519967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.519916    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:00.939930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:00.939892    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"]
Oct 30 22:12:01.951650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:01.951606    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg" event={"ID":"c25b4244-cc29-41ec-b566-19242a82d8bd","Type":"ContainerStarted","Data":"3db38be749b3e90ebc4abbb42c7b7a0cd7dfc2550e1e38b4c67e1cb72ff0de35"}
Oct 30 22:12:01.952105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:01.951676    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg" event={"ID":"c25b4244-cc29-41ec-b566-19242a82d8bd","Type":"ContainerStarted","Data":"f95d334635445689879e10b2ac0ea27832954aede86a294b7405e2ca40074587"}
Oct 30 22:12:02.075832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:02.075772    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg" podStartSLOduration=2.075749707 podStartE2EDuration="2.075749707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:02.071626267 +0000 UTC m=+1333.595769056" watchObservedRunningTime="2026-10-30 22:12:02.075749707 +0000 UTC m=+1333.599892504"
Oct 30 22:12:02.957643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:02.957604    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:02.967127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:02.967083    5031 generic.go:356] "Generic (PLEG): container finished" podID="c25b4244-cc29-41ec-b566-19242a82d8bd" containerID="3db38be749b3e90ebc4abbb42c7b7a0cd7dfc2550e1e38b4c67e1cb72ff0de35" exitCode=0
Oct 30 22:12:02.967417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:02.967396    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg" event={"ID":"c25b4244-cc29-41ec-b566-19242a82d8bd","Type":"ContainerDied","Data":"3db38be749b3e90ebc4abbb42c7b7a0cd7dfc2550e1e38b4c67e1cb72ff0de35"}
Oct 30 22:12:04.176068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.176029    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:04.364401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.364366    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-xks-ca\") pod \"c25b4244-cc29-41ec-b566-19242a82d8bd\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") "
Oct 30 22:12:04.364622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.364569    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25b4244-cc29-41ec-b566-19242a82d8bd-kube-api-access-n4d6q\" (UniqueName: \"kubernetes.io/projected/c25b4244-cc29-41ec-b566-19242a82d8bd-kube-api-access-n4d6q\") pod \"c25b4244-cc29-41ec-b566-19242a82d8bd\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") "
Oct 30 22:12:04.364622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.364607    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-openshift-ca\") pod \"c25b4244-cc29-41ec-b566-19242a82d8bd\" (UID: \"c25b4244-cc29-41ec-b566-19242a82d8bd\") "
Oct 30 22:12:04.365237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.365094    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-openshift-ca" pod "c25b4244-cc29-41ec-b566-19242a82d8bd" (UID: "c25b4244-cc29-41ec-b566-19242a82d8bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:04.365570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.365537    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:04.366846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.366819    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-xks-ca" pod "c25b4244-cc29-41ec-b566-19242a82d8bd" (UID: "c25b4244-cc29-41ec-b566-19242a82d8bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:04.366977 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.366959    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25b4244-cc29-41ec-b566-19242a82d8bd-kube-api-access-n4d6q" pod "c25b4244-cc29-41ec-b566-19242a82d8bd" (UID: "c25b4244-cc29-41ec-b566-19242a82d8bd"). InnerVolumeSpecName "kube-api-access-n4d6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:04.466674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.466635    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25b4244-cc29-41ec-b566-19242a82d8bd-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:04.466674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.466670    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4d6q\" (UniqueName: \"kubernetes.io/projected/c25b4244-cc29-41ec-b566-19242a82d8bd-kube-api-access-n4d6q\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:04.995156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.995073    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"
Oct 30 22:12:04.995156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.995102    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg" event={"ID":"c25b4244-cc29-41ec-b566-19242a82d8bd","Type":"ContainerDied","Data":"f95d334635445689879e10b2ac0ea27832954aede86a294b7405e2ca40074587"}
Oct 30 22:12:04.995156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:04.995153    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f95d334635445689879e10b2ac0ea27832954aede86a294b7405e2ca40074587"
Oct 30 22:12:06.986036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:06.986005    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:07.956291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:07.956239    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:09.619989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:09.619952    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:11.027456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.027421    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"]
Oct 30 22:12:11.028280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.028262    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c25b4244-cc29-41ec-b566-19242a82d8bd" containerName="revoke-keys"
Oct 30 22:12:11.028339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.028282    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25b4244-cc29-41ec-b566-19242a82d8bd" containerName="revoke-keys"
Oct 30 22:12:11.028430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.028420    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25b4244-cc29-41ec-b566-19242a82d8bd" containerName="revoke-keys"
Oct 30 22:12:11.032620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.032601    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.091309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.091268    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"]
Oct 30 22:12:11.134966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.134911    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:11.134966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.134910    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:11.134966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.134942    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:11.136946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.136932    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:12:11.178181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.178139    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm728\" (UniqueName: \"kubernetes.io/projected/4db64b96-3e4e-449c-a52b-4caa77792cc8-kube-api-access-wm728\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.178341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.178319    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.179066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.178451    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.280142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.280049    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm728\" (UniqueName: \"kubernetes.io/projected/4db64b96-3e4e-449c-a52b-4caa77792cc8-kube-api-access-wm728\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.280361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.280177    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.280361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.280337    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.281037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.280961    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.282848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.282824    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.344250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.344173    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm728\" (UniqueName: \"kubernetes.io/projected/4db64b96-3e4e-449c-a52b-4caa77792cc8-kube-api-access-wm728\") pod \"maas-api-revoke-sub-3a981d4c-dqznm\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.349069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.349045    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:11.750087 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:12:11.750052    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db64b96_3e4e_449c_a52b_4caa77792cc8.slice/crio-2e1a9e6ff2e0d988988f9ee0c380b380a768ef1ddb0980a4a58fe4423bb19cfc WatchSource:0}: Error finding container 2e1a9e6ff2e0d988988f9ee0c380b380a768ef1ddb0980a4a58fe4423bb19cfc: Status 404 returned error can't find the container with id 2e1a9e6ff2e0d988988f9ee0c380b380a768ef1ddb0980a4a58fe4423bb19cfc
Oct 30 22:12:11.752384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:11.752358    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"]
Oct 30 22:12:12.064530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:12.064490    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm" event={"ID":"4db64b96-3e4e-449c-a52b-4caa77792cc8","Type":"ContainerStarted","Data":"2e1a9e6ff2e0d988988f9ee0c380b380a768ef1ddb0980a4a58fe4423bb19cfc"}
Oct 30 22:12:12.956342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:12.956312    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:13.084730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:13.084581    5031 generic.go:356] "Generic (PLEG): container finished" podID="4db64b96-3e4e-449c-a52b-4caa77792cc8" containerID="dbc00935814b0cb7cbdfa9e9e809ecd2ff209e22eb1f433d85b4a3b5a1d55624" exitCode=0
Oct 30 22:12:13.086041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:13.084932    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm" event={"ID":"4db64b96-3e4e-449c-a52b-4caa77792cc8","Type":"ContainerDied","Data":"dbc00935814b0cb7cbdfa9e9e809ecd2ff209e22eb1f433d85b4a3b5a1d55624"}
Oct 30 22:12:13.182349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:13.182311    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:14.288053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.288021    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:14.433427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.433340    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-openshift-ca\") pod \"4db64b96-3e4e-449c-a52b-4caa77792cc8\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") "
Oct 30 22:12:14.433575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.433517    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-xks-ca\") pod \"4db64b96-3e4e-449c-a52b-4caa77792cc8\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") "
Oct 30 22:12:14.433690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.433673    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db64b96-3e4e-449c-a52b-4caa77792cc8-kube-api-access-wm728\" (UniqueName: \"kubernetes.io/projected/4db64b96-3e4e-449c-a52b-4caa77792cc8-kube-api-access-wm728\") pod \"4db64b96-3e4e-449c-a52b-4caa77792cc8\" (UID: \"4db64b96-3e4e-449c-a52b-4caa77792cc8\") "
Oct 30 22:12:14.433793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.433763    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-openshift-ca" pod "4db64b96-3e4e-449c-a52b-4caa77792cc8" (UID: "4db64b96-3e4e-449c-a52b-4caa77792cc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:14.434631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.434603    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:14.436044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.436020    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-xks-ca" pod "4db64b96-3e4e-449c-a52b-4caa77792cc8" (UID: "4db64b96-3e4e-449c-a52b-4caa77792cc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:14.436175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.436095    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db64b96-3e4e-449c-a52b-4caa77792cc8-kube-api-access-wm728" pod "4db64b96-3e4e-449c-a52b-4caa77792cc8" (UID: "4db64b96-3e4e-449c-a52b-4caa77792cc8"). InnerVolumeSpecName "kube-api-access-wm728". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:14.536078 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.536045    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db64b96-3e4e-449c-a52b-4caa77792cc8-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:14.536078 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:14.536072    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm728\" (UniqueName: \"kubernetes.io/projected/4db64b96-3e4e-449c-a52b-4caa77792cc8-kube-api-access-wm728\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:15.106525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:15.106487    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"
Oct 30 22:12:15.106687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:15.106524    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm" event={"ID":"4db64b96-3e4e-449c-a52b-4caa77792cc8","Type":"ContainerDied","Data":"2e1a9e6ff2e0d988988f9ee0c380b380a768ef1ddb0980a4a58fe4423bb19cfc"}
Oct 30 22:12:15.106687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:15.106603    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1a9e6ff2e0d988988f9ee0c380b380a768ef1ddb0980a4a58fe4423bb19cfc"
Oct 30 22:12:17.851149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:17.851112    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.094947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:18.094907    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:20.552073 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:12:20.552032    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:12:23.200359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:23.200329    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:24.651411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.651368    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"]
Oct 30 22:12:24.652585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.652555    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4db64b96-3e4e-449c-a52b-4caa77792cc8" containerName="revoke-keys"
Oct 30 22:12:24.652585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.652585    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db64b96-3e4e-449c-a52b-4caa77792cc8" containerName="revoke-keys"
Oct 30 22:12:24.652803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.652786    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db64b96-3e4e-449c-a52b-4caa77792cc8" containerName="revoke-keys"
Oct 30 22:12:24.657558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.657535    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.772821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.772781    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"]
Oct 30 22:12:24.819046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819001    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nrv\" (UniqueName: \"kubernetes.io/projected/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-kube-api-access-76nrv\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819063    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819114    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819159    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819184    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819244    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819465    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819549    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.819603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.819590    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921023    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921132    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921192    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921234    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921300    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921670 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921499    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921670 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921574    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921670 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921615    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.921824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.921748    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nrv\" (UniqueName: \"kubernetes.io/projected/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-kube-api-access-76nrv\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.922576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.922548    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.923253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.923179    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.923466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.923446    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.924612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.924586    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.925037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.924925    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.926910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.926885    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.945180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.945145    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.986278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.986233    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nrv\" (UniqueName: \"kubernetes.io/projected/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-kube-api-access-76nrv\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:24.986897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:24.986874    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc3fa4fe-d521-4a83-b82d-27bf5dda0f64-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-t54jj\" (UID: \"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:25.273023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:25.272934    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:25.761235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:25.761181    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"]
Oct 30 22:12:25.782825 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:12:25.782794    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3fa4fe_d521_4a83_b82d_27bf5dda0f64.slice/crio-0cdcda4545758d18baedffb8d739c9579b972eade484f852dbf65f29a9b7437c WatchSource:0}: Error finding container 0cdcda4545758d18baedffb8d739c9579b972eade484f852dbf65f29a9b7437c: Status 404 returned error can't find the container with id 0cdcda4545758d18baedffb8d739c9579b972eade484f852dbf65f29a9b7437c
Oct 30 22:12:25.785563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:25.785531    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:25.785630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:25.785602    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:25.785666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:25.785638    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:25.791384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:25.791355    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.036388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.036292    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.230807 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.230768    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" event={"ID":"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64","Type":"ContainerStarted","Data":"6b489c213582adef6d79808afe6a82e6b4893120f34e2de5169d47d54236ce9e"}
Oct 30 22:12:26.231030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.230833    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" event={"ID":"cc3fa4fe-d521-4a83-b82d-27bf5dda0f64","Type":"ContainerStarted","Data":"0cdcda4545758d18baedffb8d739c9579b972eade484f852dbf65f29a9b7437c"}
Oct 30 22:12:26.273291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.273253    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:26.274976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.274950    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:26.275109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.275000    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:26.339096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.338941    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.410479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.410423    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podStartSLOduration=2.410407039 podStartE2EDuration="2.410407039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:26.405401013 +0000 UTC m=+1357.929543786" watchObservedRunningTime="2026-10-30 22:12:26.410407039 +0000 UTC m=+1357.934549850"
Oct 30 22:12:26.648746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:26.648699    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:27.273769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:27.273730    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:27.274198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:27.273791    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:27.937553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:27.937508    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:12:28.274027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:28.273930    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:28.274027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:28.273982    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:28.446234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:28.443545    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.697991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:28.697961    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:29.277235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:29.273791    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:29.277235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:29.273857    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:30.280241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:30.279901    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:30.280241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:30.279987    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:30.426009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:30.425969    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:30.780348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:30.780296    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:12:30.829397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:30.829331    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:12:30.893373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:30.893324    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.291236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.290676    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:31.291236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.290747    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:31.529270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.524438    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.635240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.633847    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.762927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.762886    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:31.763586 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.763557    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.841251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.834316    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.841251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.834821    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:12:31.896572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.895133    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:12:31.896572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.895525    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:12:31.950816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:31.946460    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:12:32.178322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:32.176342    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:32.205231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:32.201390    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:12:32.296725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:32.295511    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:32.296725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:32.295576    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:33.275956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:33.275920    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:33.276345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:33.276312    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:34.274362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:34.274328    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:12:34.274792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:34.274420    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:12:34.352341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:34.352281    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:35.707397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:35.707363    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:35.831515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:35.831468    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:35.832148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:35.832126    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:35.833063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:35.833036    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj"
Oct 30 22:12:35.954162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:35.954100    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.202446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:36.202414    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.427990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:36.427946    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:39.529508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.529474    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:39.603569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.603523    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"]
Oct 30 22:12:39.608708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.608678    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.661369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.661321    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gmlf\" (UniqueName: \"kubernetes.io/projected/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-kube-api-access-4gmlf\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.661575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.661381    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.661638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.661589    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.672708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.672681    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:39.674647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.674624    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:39.723812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.723783    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:39.725824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.725800    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:12:39.732720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.732694    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"]
Oct 30 22:12:39.763664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.763634    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.763881 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.763858    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gmlf\" (UniqueName: \"kubernetes.io/projected/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-kube-api-access-4gmlf\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.763946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.763913    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.764441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.764419    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.766591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.766571    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.830496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.830420    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gmlf\" (UniqueName: \"kubernetes.io/projected/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-kube-api-access-4gmlf\") pod \"maas-api-revoke-sub-44278f96-jxdrv\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.859780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.859736    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"]
Oct 30 22:12:39.865517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.865488    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.904927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.904894    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"]
Oct 30 22:12:39.926994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.926946    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:39.966439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.966407    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.966617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.966446    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.966617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.966603    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f709b7-4a42-46f4-b20e-6b6a37933a23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.966757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.966627    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5vz\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-kube-api-access-dm5vz\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.966823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.966794    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.966894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.966875    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.966965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.966935    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.967085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.967055    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:39.967145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:39.967091    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.068595    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.068684    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.068785    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.068836    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.068990    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.069023    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.069143    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f709b7-4a42-46f4-b20e-6b6a37933a23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.069174    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5vz\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-kube-api-access-dm5vz\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.069278    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.069666    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.069679    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.070023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.069908    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.071192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.070873    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f709b7-4a42-46f4-b20e-6b6a37933a23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.071192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.071125    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.075517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.075468    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.090835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.090752    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.134264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.134189    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.134895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.134867    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5vz\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-kube-api-access-dm5vz\") pod \"maas-default-gateway-openshift-default-67bf986f99-c2cxq\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.182065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.182023    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:40.421505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.421200    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"]
Oct 30 22:12:40.422198 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:12:40.422164    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f1492e_8155_4281_9ff9_c1ee17fb0b34.slice/crio-bb64f82abc393e7c3eb3357b070d26c15cfba87ea2ff05cef51ee3a56f85f382 WatchSource:0}: Error finding container bb64f82abc393e7c3eb3357b070d26c15cfba87ea2ff05cef51ee3a56f85f382: Status 404 returned error can't find the container with id bb64f82abc393e7c3eb3357b070d26c15cfba87ea2ff05cef51ee3a56f85f382
Oct 30 22:12:40.595300 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:12:40.595269    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f709b7_4a42_46f4_b20e_6b6a37933a23.slice/crio-398c4d8985cecc1f1ba1149565abe13b0af86be9d6809f503f70e470ce40ee23 WatchSource:0}: Error finding container 398c4d8985cecc1f1ba1149565abe13b0af86be9d6809f503f70e470ce40ee23: Status 404 returned error can't find the container with id 398c4d8985cecc1f1ba1149565abe13b0af86be9d6809f503f70e470ce40ee23
Oct 30 22:12:40.597259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.597232    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"]
Oct 30 22:12:40.598161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.598126    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:40.598272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.598195    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:40.598313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.598280    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:40.743444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.743409    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv" event={"ID":"a6f1492e-8155-4281-9ff9-c1ee17fb0b34","Type":"ContainerStarted","Data":"bb64f82abc393e7c3eb3357b070d26c15cfba87ea2ff05cef51ee3a56f85f382"}
Oct 30 22:12:40.746284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.746251    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" event={"ID":"32f709b7-4a42-46f4-b20e-6b6a37933a23","Type":"ContainerStarted","Data":"ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63"}
Oct 30 22:12:40.746458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.746312    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" event={"ID":"32f709b7-4a42-46f4-b20e-6b6a37933a23","Type":"ContainerStarted","Data":"398c4d8985cecc1f1ba1149565abe13b0af86be9d6809f503f70e470ce40ee23"}
Oct 30 22:12:40.904258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:40.904169    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podStartSLOduration=1.904145984 podStartE2EDuration="1.904145984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:40.894674628 +0000 UTC m=+1372.418817410" watchObservedRunningTime="2026-10-30 22:12:40.904145984 +0000 UTC m=+1372.428288750"
Oct 30 22:12:41.182283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:41.182249    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:41.183512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:41.183489    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:41.183580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:41.183541    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:41.757902 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:41.757802    5031 generic.go:356] "Generic (PLEG): container finished" podID="a6f1492e-8155-4281-9ff9-c1ee17fb0b34" containerID="3d26ccdab8622074eb72c2490f6ccbceae958a533ecac72976ea07516844a575" exitCode=0
Oct 30 22:12:41.758361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:41.757896    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv" event={"ID":"a6f1492e-8155-4281-9ff9-c1ee17fb0b34","Type":"ContainerDied","Data":"3d26ccdab8622074eb72c2490f6ccbceae958a533ecac72976ea07516844a575"}
Oct 30 22:12:42.182421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:42.182382    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:42.182654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:42.182444    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:42.690572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:42.690528    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:43.039950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.039924    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:43.142362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.142328    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-openshift-ca\") pod \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") "
Oct 30 22:12:43.142549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.142457    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-kube-api-access-4gmlf\" (UniqueName: \"kubernetes.io/projected/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-kube-api-access-4gmlf\") pod \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") "
Oct 30 22:12:43.142549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.142491    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-xks-ca\") pod \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\" (UID: \"a6f1492e-8155-4281-9ff9-c1ee17fb0b34\") "
Oct 30 22:12:43.142734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.142695    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-openshift-ca" pod "a6f1492e-8155-4281-9ff9-c1ee17fb0b34" (UID: "a6f1492e-8155-4281-9ff9-c1ee17fb0b34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:43.144639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.144605    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:43.157237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.156734    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-xks-ca" pod "a6f1492e-8155-4281-9ff9-c1ee17fb0b34" (UID: "a6f1492e-8155-4281-9ff9-c1ee17fb0b34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:43.158959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.158927    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-kube-api-access-4gmlf" pod "a6f1492e-8155-4281-9ff9-c1ee17fb0b34" (UID: "a6f1492e-8155-4281-9ff9-c1ee17fb0b34"). InnerVolumeSpecName "kube-api-access-4gmlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:43.182980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.182946    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:43.183129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.183025    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:43.249953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.247231    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gmlf\" (UniqueName: \"kubernetes.io/projected/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-kube-api-access-4gmlf\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:43.249953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.247269    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f1492e-8155-4281-9ff9-c1ee17fb0b34-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:43.784347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.784320    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"
Oct 30 22:12:43.784565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.784343    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv" event={"ID":"a6f1492e-8155-4281-9ff9-c1ee17fb0b34","Type":"ContainerDied","Data":"bb64f82abc393e7c3eb3357b070d26c15cfba87ea2ff05cef51ee3a56f85f382"}
Oct 30 22:12:43.784565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:43.784397    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb64f82abc393e7c3eb3357b070d26c15cfba87ea2ff05cef51ee3a56f85f382"
Oct 30 22:12:44.182814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:44.182767    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:44.183316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:44.182845    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:44.776390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:44.776353    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:45.051731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:45.051639    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:45.183319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:45.183282    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:45.183757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:45.183360    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:46.182382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:46.182349    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:46.182548 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:46.182402    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:46.553200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:46.553116    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:47.182407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:47.182366    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:47.182664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:47.182429    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:48.182719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:48.182677    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 22:12:48.183159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:48.182749    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 22:12:49.492129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:49.492099    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:49.492674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:49.492655    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:49.493481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:49.493460    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:12:49.509819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:49.509793    5031 scope.go:122] "RemoveContainer" containerID="c81515b24a8770baf781ba1b71b4a42c402d5b3afc7800d67e193d5e9cb6ae2f"
Oct 30 22:12:49.815477 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:49.815365    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:50.421473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:50.421440    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:50.544545 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:12:50.544506    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:12:54.803809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.803769    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"]
Oct 30 22:12:54.804629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.804610    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a6f1492e-8155-4281-9ff9-c1ee17fb0b34" containerName="revoke-keys"
Oct 30 22:12:54.804629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.804628    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f1492e-8155-4281-9ff9-c1ee17fb0b34" containerName="revoke-keys"
Oct 30 22:12:54.804868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.804853    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f1492e-8155-4281-9ff9-c1ee17fb0b34" containerName="revoke-keys"
Oct 30 22:12:54.809767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.809745    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.851937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.851882    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6530c813-1262-4765-8920-e190736f90b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852030    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6530c813-1262-4765-8920-e190736f90b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852063    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvv26\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-kube-api-access-tvv26\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852299    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852411    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852500    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852557    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852620    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.852701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.852647    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.910045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.910004    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"]
Oct 30 22:12:54.954031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.953975    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954094    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954167    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954224    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954257    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954307    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6530c813-1262-4765-8920-e190736f90b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954391    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6530c813-1262-4765-8920-e190736f90b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954431    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvv26\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-kube-api-access-tvv26\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954576    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.954745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.954647    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.955295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.955272    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.955433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.955410    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6530c813-1262-4765-8920-e190736f90b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.955535 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.955516    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.956069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.955980    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.959163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.959119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6530c813-1262-4765-8920-e190736f90b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.960503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.960475    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:54.995992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:54.995955    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:55.018681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.018642    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:55.018882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.018858    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvv26\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-kube-api-access-tvv26\") pod \"maas-default-gateway-openshift-default-67bf986f99-mzffc\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:55.126986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.126955    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:55.579011 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:12:55.578972    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6530c813_1262_4765_8920_e190736f90b2.slice/crio-250601ffc31b136d9775fa4257d28d50ab0b762d10e46dcf8cc527836cfbab2a WatchSource:0}: Error finding container 250601ffc31b136d9775fa4257d28d50ab0b762d10e46dcf8cc527836cfbab2a: Status 404 returned error can't find the container with id 250601ffc31b136d9775fa4257d28d50ab0b762d10e46dcf8cc527836cfbab2a
Oct 30 22:12:55.580113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.580088    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"]
Oct 30 22:12:55.581800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.581765    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:55.581892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.581831    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:55.581892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.581860    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:55.945723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.945683    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" event={"ID":"6530c813-1262-4765-8920-e190736f90b2","Type":"ContainerStarted","Data":"2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35"}
Oct 30 22:12:55.946267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:55.945745    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" event={"ID":"6530c813-1262-4765-8920-e190736f90b2","Type":"ContainerStarted","Data":"250601ffc31b136d9775fa4257d28d50ab0b762d10e46dcf8cc527836cfbab2a"}
Oct 30 22:12:56.074067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:56.074009    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podStartSLOduration=2.07399484 podStartE2EDuration="2.07399484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:56.07106 +0000 UTC m=+1387.595202783" watchObservedRunningTime="2026-10-30 22:12:56.07399484 +0000 UTC m=+1387.598137604"
Oct 30 22:12:56.127823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:56.127789    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:12:56.129624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:56.129591    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:12:56.129805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:56.129653    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:12:57.129238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:57.128630    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:12:57.129238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:57.128709    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:12:58.131709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:58.131662    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:12:58.132178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:58.131756    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:12:58.902146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:58.902105    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:59.132774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:59.132727    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:12:59.133498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:12:59.133465    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:13:00.135662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:00.135623    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:13:00.136735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:00.136311    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:13:00.390296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:00.388545    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:01.129320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:01.128613    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:13:01.129320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:01.128673    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:13:02.128189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:02.128146    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:13:02.128866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:02.128825    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:13:03.127516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:03.127464    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mzffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:13:03.127777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:03.127534    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:13:04.289490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:04.289450    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:13:04.290040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:04.290020    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:13:04.291185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:04.291164    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:13:05.629137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:05.629101    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:11.630549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:11.628337    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.800243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:11.776357    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:11.800243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:11.778148    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:13:11.800243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:11.778260    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:13:11.963303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:11.950353    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.007240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.003279    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.187402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.187352    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.243442 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.243343    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.245630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.245579    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.483907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.483863    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.624362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.624310    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.699235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.693570    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.777241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.776776    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:13:12.813233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.812284    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:13:12.813699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.813625    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:13:12.820247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.814659    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:13:12.820247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.815249    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:13:12.844445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.844396    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:12.845356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.845330    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:13:12.858363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:12.858322    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": context deadline exceeded"
Oct 30 22:13:13.244330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.219380    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.244330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.220130    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:13:13.291019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.290883    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.292711 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.292677    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:13:13.509505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.493388    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:13:13.509505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.506939    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:13:13.803296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.795108    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.835466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.835391    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.870832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.847300    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.870832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.847624    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:13.955252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:13.945600    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:13:14.196241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:14.188272    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Oct 30 22:13:14.427242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:14.426679    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 22:13:14.554480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:14.554354    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:14.867237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:14.852708    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:13:14.867237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:14.853127    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:14.867237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:14.854766    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:13:15.238824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.238658    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:15.375468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.374379    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:15.535678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.527533    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:15.666236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.665136    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c4r4m" podUID="ed248874-db33-429b-b509-2a75e3e72304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:15.752247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.723878    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:15.752247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.728278    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded"
Oct 30 22:13:15.817292 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:13:15.816706    5031 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.124s"
Oct 30 22:13:15.817292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.816887    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:13:15.817292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.816932    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:13:15.859067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.859023    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:15.863475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.859266    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:13:15.863822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.859316    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:15.892824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.892787    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:13:15.916044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:15.915913    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:13:16.008272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:16.008177    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:13:16.052531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:16.052498    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:13:17.593856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:17.593823    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:19.034617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.034565    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"]
Oct 30 22:13:19.040341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.040304    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.101116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.101075    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"]
Oct 30 22:13:19.143136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.143102    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:19.143431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.143188    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:13:19.143551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.143403    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:19.145421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.145400    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:19.155081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.155042    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwnf\" (UniqueName: \"kubernetes.io/projected/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-kube-api-access-dzwnf\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.155489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.155469    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.155596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.155553    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.257420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.257384    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.257640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.257479    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.257724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.257664    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwnf\" (UniqueName: \"kubernetes.io/projected/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-kube-api-access-dzwnf\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.258169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.258124    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.260834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.260805    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.322350 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.322265    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwnf\" (UniqueName: \"kubernetes.io/projected/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-kube-api-access-dzwnf\") pod \"maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.359457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.359425    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:19.791153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.791112    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"]
Oct 30 22:13:19.792380 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:13:19.792346    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef7cdeb_bbbd_4e0d_9dde_a06b76e1743f.slice/crio-29d411f5f595cc89f2d937ec8e39ec4f32b200c87fbf766a10f600dc843e9cab WatchSource:0}: Error finding container 29d411f5f595cc89f2d937ec8e39ec4f32b200c87fbf766a10f600dc843e9cab: Status 404 returned error can't find the container with id 29d411f5f595cc89f2d937ec8e39ec4f32b200c87fbf766a10f600dc843e9cab
Oct 30 22:13:19.883712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:19.883667    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff" event={"ID":"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f","Type":"ContainerStarted","Data":"29d411f5f595cc89f2d937ec8e39ec4f32b200c87fbf766a10f600dc843e9cab"}
Oct 30 22:13:20.607025 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:13:20.606981    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:13:20.897761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:20.897658    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff" event={"ID":"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f","Type":"ContainerStarted","Data":"e003e3907f35f3f8bde71dd106ad899aca448f4e8682cba275bea383c790f177"}
Oct 30 22:13:21.020070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:21.020003    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff" podStartSLOduration=3.019989062 podStartE2EDuration="3.019989062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:21.018022604 +0000 UTC m=+1412.542165388" watchObservedRunningTime="2026-10-30 22:13:21.019989062 +0000 UTC m=+1412.544131883"
Oct 30 22:13:21.914083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:21.914042    5031 generic.go:356] "Generic (PLEG): container finished" podID="5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" containerID="e003e3907f35f3f8bde71dd106ad899aca448f4e8682cba275bea383c790f177" exitCode=0
Oct 30 22:13:21.914651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:21.914101    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff" event={"ID":"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f","Type":"ContainerDied","Data":"e003e3907f35f3f8bde71dd106ad899aca448f4e8682cba275bea383c790f177"}
Oct 30 22:13:23.135947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.135915    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:23.244945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.244857    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:23.342983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.342950    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-kube-api-access-dzwnf\" (UniqueName: \"kubernetes.io/projected/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-kube-api-access-dzwnf\") pod \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") "
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.343149    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-openshift-ca\") pod \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") "
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.343181    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-xks-ca\") pod \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\" (UID: \"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f\") "
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.343516    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-openshift-ca" pod "5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" (UID: "5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.344152    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.346182    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-kube-api-access-dzwnf" pod "5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" (UID: "5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f"). InnerVolumeSpecName "kube-api-access-dzwnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.346297    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-xks-ca" pod "5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" (UID: "5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.446942    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:23.448027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.446977    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwnf\" (UniqueName: \"kubernetes.io/projected/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f-kube-api-access-dzwnf\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:23.994226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.994175    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"
Oct 30 22:13:23.994434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.994174    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff" event={"ID":"5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f","Type":"ContainerDied","Data":"29d411f5f595cc89f2d937ec8e39ec4f32b200c87fbf766a10f600dc843e9cab"}
Oct 30 22:13:23.994434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.994374    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29d411f5f595cc89f2d937ec8e39ec4f32b200c87fbf766a10f600dc843e9cab"
Oct 30 22:13:23.996790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.996758    5031 generic.go:356] "Generic (PLEG): container finished" podID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerID="df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325" exitCode=0
Oct 30 22:13:23.996956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.996796    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" event={"ID":"1f6a805d-46e7-499f-b443-fde8acc5863a","Type":"ContainerDied","Data":"df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325"}
Oct 30 22:13:23.997416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:23.997394    5031 scope.go:122] "RemoveContainer" containerID="df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325"
Oct 30 22:13:24.001265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:24.001195    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:24.001409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:24.001312    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:24.001409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:24.001380    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:24.838240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:24.832699    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:25.022301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:25.021417    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" event={"ID":"1f6a805d-46e7-499f-b443-fde8acc5863a","Type":"ContainerStarted","Data":"ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8"}
Oct 30 22:13:25.540134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:25.540088    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:13:25.542059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:25.542023    5031 patch_prober.go:29] interesting pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:13:25.542304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:25.542088    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:13:26.541282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:26.540777    5031 patch_prober.go:29] interesting pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:13:26.541282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:26.540849    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:13:27.241263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:27.240448    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:13:27.539917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:27.539826    5031 patch_prober.go:29] interesting pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:13:27.540162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:27.540132    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:13:28.497049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.497007    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"]
Oct 30 22:13:28.497857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.497840    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" containerName="revoke-keys"
Oct 30 22:13:28.497857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.497857    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" containerName="revoke-keys"
Oct 30 22:13:28.498053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.498041    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" containerName="revoke-keys"
Oct 30 22:13:28.501456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.501433    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.543731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.543695    5031 patch_prober.go:29] interesting pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:13:28.544022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.543995    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:13:28.561112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.561074    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"]
Oct 30 22:13:28.584881 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.584848    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:28.584881 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.584856    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:28.587775 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.587730    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.587936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.587853    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvqj\" (UniqueName: \"kubernetes.io/projected/cfe45f2c-e313-4a02-a850-83d833003308-kube-api-access-6lvqj\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.587936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.587888    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.603467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.603432    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:28.603701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.603685    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:13:28.689103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.689062    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"]
Oct 30 22:13:28.690269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.690198    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.690423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.690326    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvqj\" (UniqueName: \"kubernetes.io/projected/cfe45f2c-e313-4a02-a850-83d833003308-kube-api-access-6lvqj\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.690423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.690369    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.691667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.691637    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.694175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.694148    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.695240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.695195    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.756151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.756115    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"]
Oct 30 22:13:28.756472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.756344    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvqj\" (UniqueName: \"kubernetes.io/projected/cfe45f2c-e313-4a02-a850-83d833003308-kube-api-access-6lvqj\") pod \"maas-api-revoke-sub-e5a8cb80-2xfnp\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.793286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.793250    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.793510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.793396    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjw49\" (UniqueName: \"kubernetes.io/projected/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-kube-api-access-wjw49\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.793795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.793756    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.858910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.858873    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:28.895556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.895505    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.895774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.895742    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.895907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.895888    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjw49\" (UniqueName: \"kubernetes.io/projected/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-kube-api-access-wjw49\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.896731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.896704    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.898402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.898364    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:28.965804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:28.965769    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjw49\" (UniqueName: \"kubernetes.io/projected/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-kube-api-access-wjw49\") pod \"maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:29.066308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:29.066191    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:29.295184 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:13:29.295151    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe45f2c_e313_4a02_a850_83d833003308.slice/crio-c1bd32328c753f3cb21925d1f1cdcd9a0fa8962a92bd973b8468583f3546acb4 WatchSource:0}: Error finding container c1bd32328c753f3cb21925d1f1cdcd9a0fa8962a92bd973b8468583f3546acb4: Status 404 returned error can't find the container with id c1bd32328c753f3cb21925d1f1cdcd9a0fa8962a92bd973b8468583f3546acb4
Oct 30 22:13:29.295824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:29.295798    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"]
Oct 30 22:13:29.533453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:29.532309    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"]
Oct 30 22:13:29.545423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:29.545397    5031 patch_prober.go:29] interesting pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:13:29.545506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:29.545443    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:13:29.856088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:29.856051    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"]
Oct 30 22:13:29.963455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:29.963427    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"]
Oct 30 22:13:30.103682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.103645    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp" event={"ID":"cfe45f2c-e313-4a02-a850-83d833003308","Type":"ContainerStarted","Data":"329a7fdf468dfd0270504ab7f3b1222f90f2035d1aa01c4fea7f2f09e4dc905c"}
Oct 30 22:13:30.103970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.103715    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp" event={"ID":"cfe45f2c-e313-4a02-a850-83d833003308","Type":"ContainerStarted","Data":"c1bd32328c753f3cb21925d1f1cdcd9a0fa8962a92bd973b8468583f3546acb4"}
Oct 30 22:13:30.105583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.105556    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp" event={"ID":"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b","Type":"ContainerStarted","Data":"018d59d88e1c5d811e79af9cf9ff2153be4c8de162dac3283fb3a4597d60c1db"}
Oct 30 22:13:30.105746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.105596    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp" event={"ID":"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b","Type":"ContainerStarted","Data":"18828768a3996873ba244249bfa9710f2d823bc2efa997c322a26f3009f99e35"}
Oct 30 22:13:30.253139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.253023    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp" podStartSLOduration=2.2530086049999998 podStartE2EDuration="2.253008605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:30.245940267 +0000 UTC m=+1421.770083052" watchObservedRunningTime="2026-10-30 22:13:30.253008605 +0000 UTC m=+1421.777151371"
Oct 30 22:13:30.318918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.318423    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:30.624643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.624608    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:13:30.629842 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.629815    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:13:30.696370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.696338    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:30.754696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:30.754638    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp" podStartSLOduration=2.754621867 podStartE2EDuration="2.754621867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:30.382802166 +0000 UTC m=+1421.906944956" watchObservedRunningTime="2026-10-30 22:13:30.754621867 +0000 UTC m=+1422.278764662"
Oct 30 22:13:31.121371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:31.121326    5031 generic.go:356] "Generic (PLEG): container finished" podID="cfe45f2c-e313-4a02-a850-83d833003308" containerID="329a7fdf468dfd0270504ab7f3b1222f90f2035d1aa01c4fea7f2f09e4dc905c" exitCode=0
Oct 30 22:13:31.121560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:31.121415    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp" event={"ID":"cfe45f2c-e313-4a02-a850-83d833003308","Type":"ContainerDied","Data":"329a7fdf468dfd0270504ab7f3b1222f90f2035d1aa01c4fea7f2f09e4dc905c"}
Oct 30 22:13:31.123527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:31.123502    5031 generic.go:356] "Generic (PLEG): container finished" podID="8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" containerID="018d59d88e1c5d811e79af9cf9ff2153be4c8de162dac3283fb3a4597d60c1db" exitCode=0
Oct 30 22:13:31.123682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:31.123588    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp" event={"ID":"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b","Type":"ContainerDied","Data":"018d59d88e1c5d811e79af9cf9ff2153be4c8de162dac3283fb3a4597d60c1db"}
Oct 30 22:13:32.684429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.684392    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:32.690436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.689863    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:32.773712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.773623    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-xks-ca\") pod \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") "
Oct 30 22:13:32.773712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.773698    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-openshift-ca\") pod \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") "
Oct 30 22:13:32.773894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.773772    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-openshift-ca\") pod \"cfe45f2c-e313-4a02-a850-83d833003308\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") "
Oct 30 22:13:32.773894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.773801    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-kube-api-access-wjw49\" (UniqueName: \"kubernetes.io/projected/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-kube-api-access-wjw49\") pod \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\" (UID: \"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b\") "
Oct 30 22:13:32.773996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.773922    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe45f2c-e313-4a02-a850-83d833003308-kube-api-access-6lvqj\" (UniqueName: \"kubernetes.io/projected/cfe45f2c-e313-4a02-a850-83d833003308-kube-api-access-6lvqj\") pod \"cfe45f2c-e313-4a02-a850-83d833003308\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") "
Oct 30 22:13:32.774049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.773997    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-xks-ca\") pod \"cfe45f2c-e313-4a02-a850-83d833003308\" (UID: \"cfe45f2c-e313-4a02-a850-83d833003308\") "
Oct 30 22:13:32.774141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.774105    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-openshift-ca" pod "8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" (UID: "8324b0c6-c8ab-44f9-a7f0-70eb39b1320b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:32.774251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.774138    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-openshift-ca" pod "cfe45f2c-e313-4a02-a850-83d833003308" (UID: "cfe45f2c-e313-4a02-a850-83d833003308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:32.774755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.774676    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:32.774755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.774700    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:32.776606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.776559    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe45f2c-e313-4a02-a850-83d833003308-kube-api-access-6lvqj" pod "cfe45f2c-e313-4a02-a850-83d833003308" (UID: "cfe45f2c-e313-4a02-a850-83d833003308"). InnerVolumeSpecName "kube-api-access-6lvqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:32.776728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.776662    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-xks-ca" pod "cfe45f2c-e313-4a02-a850-83d833003308" (UID: "cfe45f2c-e313-4a02-a850-83d833003308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:32.776904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.776877    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-xks-ca" pod "8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" (UID: "8324b0c6-c8ab-44f9-a7f0-70eb39b1320b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:32.777013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.776910    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-kube-api-access-wjw49" pod "8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" (UID: "8324b0c6-c8ab-44f9-a7f0-70eb39b1320b"). InnerVolumeSpecName "kube-api-access-wjw49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:32.877228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.877173    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:32.877410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.877234    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjw49\" (UniqueName: \"kubernetes.io/projected/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b-kube-api-access-wjw49\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:32.877410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.877251    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvqj\" (UniqueName: \"kubernetes.io/projected/cfe45f2c-e313-4a02-a850-83d833003308-kube-api-access-6lvqj\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:32.877410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:32.877266    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe45f2c-e313-4a02-a850-83d833003308-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:33.240712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:33.240675    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp" event={"ID":"cfe45f2c-e313-4a02-a850-83d833003308","Type":"ContainerDied","Data":"c1bd32328c753f3cb21925d1f1cdcd9a0fa8962a92bd973b8468583f3546acb4"}
Oct 30 22:13:33.240712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:33.240696    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"
Oct 30 22:13:33.240991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:33.240741    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1bd32328c753f3cb21925d1f1cdcd9a0fa8962a92bd973b8468583f3546acb4"
Oct 30 22:13:33.242582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:33.242554    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"
Oct 30 22:13:33.242739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:33.242596    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp" event={"ID":"8324b0c6-c8ab-44f9-a7f0-70eb39b1320b","Type":"ContainerDied","Data":"18828768a3996873ba244249bfa9710f2d823bc2efa997c322a26f3009f99e35"}
Oct 30 22:13:33.242739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:33.242641    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18828768a3996873ba244249bfa9710f2d823bc2efa997c322a26f3009f99e35"
Oct 30 22:13:34.861384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:34.861343    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245" podUID="7ab19abc-fc51-48ca-8f50-f65f37837d6b" containerName="payload-processing" containerID="cri-o://6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c" gracePeriod=25
Oct 30 22:13:34.964859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:34.964809    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf" podUID="f3b930a7-116d-4fd2-ae48-d9d77b1ad933" containerName="payload-pre-processing" containerID="cri-o://7d7b8ae03a83f3158e01445295e48e32b77fb88ba95c48ae4d243705791b218b" gracePeriod=25
Oct 30 22:13:35.379516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.364139    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:13:35.492942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.474839    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab19abc-fc51-48ca-8f50-f65f37837d6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab19abc-fc51-48ca-8f50-f65f37837d6b-plugins-config-volume\") pod \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") "
Oct 30 22:13:35.492942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.475035    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab19abc-fc51-48ca-8f50-f65f37837d6b-kube-api-access-49twr\" (UniqueName: \"kubernetes.io/projected/7ab19abc-fc51-48ca-8f50-f65f37837d6b-kube-api-access-49twr\") pod \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\" (UID: \"7ab19abc-fc51-48ca-8f50-f65f37837d6b\") "
Oct 30 22:13:35.492942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.490689    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab19abc-fc51-48ca-8f50-f65f37837d6b-plugins-config-volume" pod "7ab19abc-fc51-48ca-8f50-f65f37837d6b" (UID: "7ab19abc-fc51-48ca-8f50-f65f37837d6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:35.592339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.592298    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab19abc-fc51-48ca-8f50-f65f37837d6b-kube-api-access-49twr" pod "7ab19abc-fc51-48ca-8f50-f65f37837d6b" (UID: "7ab19abc-fc51-48ca-8f50-f65f37837d6b"). InnerVolumeSpecName "kube-api-access-49twr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:35.602415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.602379    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab19abc-fc51-48ca-8f50-f65f37837d6b-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.602415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.602416    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49twr\" (UniqueName: \"kubernetes.io/projected/7ab19abc-fc51-48ca-8f50-f65f37837d6b-kube-api-access-49twr\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.646326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.646287    5031 generic.go:356] "Generic (PLEG): container finished" podID="7ab19abc-fc51-48ca-8f50-f65f37837d6b" containerID="6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c" exitCode=0
Oct 30 22:13:35.646637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.646616    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"
Oct 30 22:13:35.646893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.646477    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245" event={"ID":"7ab19abc-fc51-48ca-8f50-f65f37837d6b","Type":"ContainerDied","Data":"6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c"}
Oct 30 22:13:35.647000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.646952    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245" event={"ID":"7ab19abc-fc51-48ca-8f50-f65f37837d6b","Type":"ContainerDied","Data":"0c31d5d41199ac048ba94c23955c6919175156cebe2ca200e94fcf88671cb63d"}
Oct 30 22:13:35.647068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.646997    5031 scope.go:122] "RemoveContainer" containerID="6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c"
Oct 30 22:13:35.655604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.654190    5031 generic.go:356] "Generic (PLEG): container finished" podID="f3b930a7-116d-4fd2-ae48-d9d77b1ad933" containerID="7d7b8ae03a83f3158e01445295e48e32b77fb88ba95c48ae4d243705791b218b" exitCode=0
Oct 30 22:13:35.655604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.654362    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf" event={"ID":"f3b930a7-116d-4fd2-ae48-d9d77b1ad933","Type":"ContainerDied","Data":"7d7b8ae03a83f3158e01445295e48e32b77fb88ba95c48ae4d243705791b218b"}
Oct 30 22:13:35.663363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.662664    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:13:35.668991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.668966    5031 scope.go:122] "RemoveContainer" containerID="6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c"
Oct 30 22:13:35.669367 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:13:35.669345    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c\": container with ID starting with 6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c not found: ID does not exist" containerID="6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c"
Oct 30 22:13:35.669444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.669377    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c"} err="failed to get container status \"6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c\": rpc error: code = NotFound desc = could not find container \"6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c\": container with ID starting with 6ff81451d2e216c7c9bb9daf6f0e1024d24d314798ab101c8369f1a91400289c not found: ID does not exist"
Oct 30 22:13:35.702982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.702952    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-kube-api-access-pw8mh\" (UniqueName: \"kubernetes.io/projected/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-kube-api-access-pw8mh\") pod \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") "
Oct 30 22:13:35.703270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.703250    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-plugins-config-volume\") pod \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\" (UID: \"f3b930a7-116d-4fd2-ae48-d9d77b1ad933\") "
Oct 30 22:13:35.703647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.703617    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-plugins-config-volume" pod "f3b930a7-116d-4fd2-ae48-d9d77b1ad933" (UID: "f3b930a7-116d-4fd2-ae48-d9d77b1ad933"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:35.704054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.704036    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.705484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.705461    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-kube-api-access-pw8mh" pod "f3b930a7-116d-4fd2-ae48-d9d77b1ad933" (UID: "f3b930a7-116d-4fd2-ae48-d9d77b1ad933"). InnerVolumeSpecName "kube-api-access-pw8mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:35.806352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.806188    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8mh\" (UniqueName: \"kubernetes.io/projected/f3b930a7-116d-4fd2-ae48-d9d77b1ad933-kube-api-access-pw8mh\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.841059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.841013    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"]
Oct 30 22:13:35.847522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:35.847490    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245"]
Oct 30 22:13:36.305990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:36.305950    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:36.649551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:36.649506    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab19abc-fc51-48ca-8f50-f65f37837d6b" path="/var/lib/kubelet/pods/7ab19abc-fc51-48ca-8f50-f65f37837d6b/volumes"
Oct 30 22:13:36.666818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:36.666783    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"
Oct 30 22:13:36.666818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:36.666801    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf" event={"ID":"f3b930a7-116d-4fd2-ae48-d9d77b1ad933","Type":"ContainerDied","Data":"5867fe5d78d0ff26fc94edc97896b339768182b6e45efb63ce86ddc75e1892ea"}
Oct 30 22:13:36.667079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:36.666868    5031 scope.go:122] "RemoveContainer" containerID="7d7b8ae03a83f3158e01445295e48e32b77fb88ba95c48ae4d243705791b218b"
Oct 30 22:13:36.849404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:36.849364    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"]
Oct 30 22:13:36.860316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:36.860277    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf"]
Oct 30 22:13:38.652054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:38.652011    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b930a7-116d-4fd2-ae48-d9d77b1ad933" path="/var/lib/kubelet/pods/f3b930a7-116d-4fd2-ae48-d9d77b1ad933/volumes"
Oct 30 22:13:39.220471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:39.220425    5031 patch_prober.go:29] interesting pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:13:39.220782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:39.220740    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:39.363898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:39.363859    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:39.599882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:39.599823    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:39.829958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:39.829885    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:41.483840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.483792    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"]
Oct 30 22:13:41.484640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484623    5031 cpu_manager.go:414] "containerMap: removing container" podUID="cfe45f2c-e313-4a02-a850-83d833003308" containerName="revoke-keys"
Oct 30 22:13:41.484692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484642    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe45f2c-e313-4a02-a850-83d833003308" containerName="revoke-keys"
Oct 30 22:13:41.484692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484673    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f3b930a7-116d-4fd2-ae48-d9d77b1ad933" containerName="payload-pre-processing"
Oct 30 22:13:41.484692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484678    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b930a7-116d-4fd2-ae48-d9d77b1ad933" containerName="payload-pre-processing"
Oct 30 22:13:41.484790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484702    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" containerName="revoke-keys"
Oct 30 22:13:41.484790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484707    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" containerName="revoke-keys"
Oct 30 22:13:41.484790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484716    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7ab19abc-fc51-48ca-8f50-f65f37837d6b" containerName="payload-processing"
Oct 30 22:13:41.484790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484720    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab19abc-fc51-48ca-8f50-f65f37837d6b" containerName="payload-processing"
Oct 30 22:13:41.484931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484846    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" containerName="revoke-keys"
Oct 30 22:13:41.484931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484859    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab19abc-fc51-48ca-8f50-f65f37837d6b" containerName="payload-processing"
Oct 30 22:13:41.484931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484876    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe45f2c-e313-4a02-a850-83d833003308" containerName="revoke-keys"
Oct 30 22:13:41.484931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.484890    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b930a7-116d-4fd2-ae48-d9d77b1ad933" containerName="payload-pre-processing"
Oct 30 22:13:41.613138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.613093    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"]
Oct 30 22:13:41.613402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.613224    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.678886 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.678831    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbxr\" (UniqueName: \"kubernetes.io/projected/03fc08e7-13f0-482e-a150-1af21e3e8d23-kube-api-access-zwbxr\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.679129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.678902    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.679129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.679056    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.725552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.725513    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:41.725552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.725522    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:41.725819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.725608    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:41.725819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.725683    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:13:41.780605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.780518    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbxr\" (UniqueName: \"kubernetes.io/projected/03fc08e7-13f0-482e-a150-1af21e3e8d23-kube-api-access-zwbxr\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.780798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.780608    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.780798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.780710    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.781424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.781403    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.783841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.783812    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:41.847749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:41.847709    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbxr\" (UniqueName: \"kubernetes.io/projected/03fc08e7-13f0-482e-a150-1af21e3e8d23-kube-api-access-zwbxr\") pod \"maas-api-revoke-keys-e2e-ait-643a82de-d4sff\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:42.051198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:42.051116    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:42.114973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:42.114940    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:42.556891 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:13:42.556853    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fc08e7_13f0_482e_a150_1af21e3e8d23.slice/crio-527a67f0471939645689acd8ddc814bfd4388d860e1b0a306439d10438f4a4cf WatchSource:0}: Error finding container 527a67f0471939645689acd8ddc814bfd4388d860e1b0a306439d10438f4a4cf: Status 404 returned error can't find the container with id 527a67f0471939645689acd8ddc814bfd4388d860e1b0a306439d10438f4a4cf
Oct 30 22:13:42.560152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:42.560121    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"]
Oct 30 22:13:42.816165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:42.816068    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff" event={"ID":"03fc08e7-13f0-482e-a150-1af21e3e8d23","Type":"ContainerStarted","Data":"527a67f0471939645689acd8ddc814bfd4388d860e1b0a306439d10438f4a4cf"}
Oct 30 22:13:43.833667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:43.833564    5031 generic.go:356] "Generic (PLEG): container finished" podID="03fc08e7-13f0-482e-a150-1af21e3e8d23" containerID="11c857ad1d43bd4c03933f8fc7450c64b80b714fe33907c0697b8d832fa442e5" exitCode=0
Oct 30 22:13:43.834384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:43.833657    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff" event={"ID":"03fc08e7-13f0-482e-a150-1af21e3e8d23","Type":"ContainerDied","Data":"11c857ad1d43bd4c03933f8fc7450c64b80b714fe33907c0697b8d832fa442e5"}
Oct 30 22:13:45.101813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.101735    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:45.146726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.146685    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-openshift-ca\") pod \"03fc08e7-13f0-482e-a150-1af21e3e8d23\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") "
Oct 30 22:13:45.147000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.146981    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-xks-ca\") pod \"03fc08e7-13f0-482e-a150-1af21e3e8d23\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") "
Oct 30 22:13:45.147104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.147074    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-openshift-ca" pod "03fc08e7-13f0-482e-a150-1af21e3e8d23" (UID: "03fc08e7-13f0-482e-a150-1af21e3e8d23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:45.147224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.147191    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fc08e7-13f0-482e-a150-1af21e3e8d23-kube-api-access-zwbxr\" (UniqueName: \"kubernetes.io/projected/03fc08e7-13f0-482e-a150-1af21e3e8d23-kube-api-access-zwbxr\") pod \"03fc08e7-13f0-482e-a150-1af21e3e8d23\" (UID: \"03fc08e7-13f0-482e-a150-1af21e3e8d23\") "
Oct 30 22:13:45.147929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.147906    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:45.169277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.169239    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-xks-ca" pod "03fc08e7-13f0-482e-a150-1af21e3e8d23" (UID: "03fc08e7-13f0-482e-a150-1af21e3e8d23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:45.169416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.169392    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fc08e7-13f0-482e-a150-1af21e3e8d23-kube-api-access-zwbxr" pod "03fc08e7-13f0-482e-a150-1af21e3e8d23" (UID: "03fc08e7-13f0-482e-a150-1af21e3e8d23"). InnerVolumeSpecName "kube-api-access-zwbxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:45.252432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.250341    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbxr\" (UniqueName: \"kubernetes.io/projected/03fc08e7-13f0-482e-a150-1af21e3e8d23-kube-api-access-zwbxr\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:45.252432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.250376    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fc08e7-13f0-482e-a150-1af21e3e8d23-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:45.862191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.862160    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"
Oct 30 22:13:45.862560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.862201    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff" event={"ID":"03fc08e7-13f0-482e-a150-1af21e3e8d23","Type":"ContainerDied","Data":"527a67f0471939645689acd8ddc814bfd4388d860e1b0a306439d10438f4a4cf"}
Oct 30 22:13:45.862703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:45.862586    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="527a67f0471939645689acd8ddc814bfd4388d860e1b0a306439d10438f4a4cf"
Oct 30 22:13:48.315922 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:48.315878    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:48.537786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:48.537748    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:49.903889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:49.903853    5031 generic.go:356] "Generic (PLEG): container finished" podID="9aabc72e-c473-4680-9422-c32ddd053f82" containerID="3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4" exitCode=0
Oct 30 22:13:49.904374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:49.903928    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" event={"ID":"9aabc72e-c473-4680-9422-c32ddd053f82","Type":"ContainerDied","Data":"3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4"}
Oct 30 22:13:49.904481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:49.904464    5031 scope.go:122] "RemoveContainer" containerID="3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4"
Oct 30 22:13:49.906318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:49.906281    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:49.906441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:49.906376    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:49.906502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:49.906437    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:50.549022 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:13:50.548984    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:13:50.952822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:50.952781    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" event={"ID":"9aabc72e-c473-4680-9422-c32ddd053f82","Type":"ContainerStarted","Data":"a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de"}
Oct 30 22:13:51.150368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:51.150335    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:13:51.151158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:51.151127    5031 patch_prober.go:29] interesting pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:13:51.151356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:51.151188    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:13:51.591899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:51.591860    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:13:51.896858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:51.896817    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"]
Oct 30 22:13:52.151026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:52.150946    5031 patch_prober.go:29] interesting pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:13:52.151026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:52.151004    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:13:52.444364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:52.443791    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:52.668295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:52.668260    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:53.151388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:53.151345    5031 patch_prober.go:29] interesting pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:13:53.151905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:53.151421    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:13:53.178942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:53.178913    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:13:53.535299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:53.524284    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:54.079360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:54.079322    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"]
Oct 30 22:13:54.104848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:54.104810    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14363982-rx8gj"]
Oct 30 22:13:54.176950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:54.176911    5031 patch_prober.go:29] interesting pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:13:54.177462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:54.176995    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:13:54.650955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:54.650903    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92738c9a-624f-4658-8071-40de3b792487" path="/var/lib/kubelet/pods/92738c9a-624f-4658-8071-40de3b792487/volumes"
Oct 30 22:13:55.150847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:55.150807    5031 patch_prober.go:29] interesting pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:13:55.151045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:55.150890    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:13:56.150547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:56.150507    5031 patch_prober.go:29] interesting pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:13:56.151021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:56.150571    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:13:56.897695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:56.897648    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55" podUID="e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" containerName="payload-processing" containerID="cri-o://f888b77b3a04074c206da4c8db89e6921ff90da255ea7ba0c72b7c5b6e95a754" gracePeriod=25
Oct 30 22:13:57.159799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.159694    5031 generic.go:356] "Generic (PLEG): container finished" podID="e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" containerID="f888b77b3a04074c206da4c8db89e6921ff90da255ea7ba0c72b7c5b6e95a754" exitCode=0
Oct 30 22:13:57.159799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.159773    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55" event={"ID":"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f","Type":"ContainerDied","Data":"f888b77b3a04074c206da4c8db89e6921ff90da255ea7ba0c72b7c5b6e95a754"}
Oct 30 22:13:57.160426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.160312    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:13:57.161418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.161398    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:13:57.216689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.216658    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:13:57.275582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.275542    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-plugins-config-volume\") pod \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") "
Oct 30 22:13:57.275873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.275854    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-kube-api-access-2cqt9\" (UniqueName: \"kubernetes.io/projected/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-kube-api-access-2cqt9\") pod \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\" (UID: \"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f\") "
Oct 30 22:13:57.276119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.276090    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-plugins-config-volume" pod "e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" (UID: "e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:57.276956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.276803    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.278705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.278672    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-kube-api-access-2cqt9" pod "e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" (UID: "e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f"). InnerVolumeSpecName "kube-api-access-2cqt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:57.378844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:57.378804    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqt9\" (UniqueName: \"kubernetes.io/projected/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f-kube-api-access-2cqt9\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:58.173615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:58.173585    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"
Oct 30 22:13:58.174138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:58.173605    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55" event={"ID":"e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f","Type":"ContainerDied","Data":"99b125c6cf99285fbb5ecdb08c0cf99ad62bf62af8f5759e9afb45579db36446"}
Oct 30 22:13:58.174138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:58.173685    5031 scope.go:122] "RemoveContainer" containerID="f888b77b3a04074c206da4c8db89e6921ff90da255ea7ba0c72b7c5b6e95a754"
Oct 30 22:13:58.365564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:58.365526    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"]
Oct 30 22:13:58.415352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:58.415320    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55"]
Oct 30 22:13:58.651914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:58.651874    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" path="/var/lib/kubelet/pods/e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f/volumes"
Oct 30 22:13:58.792522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:13:58.792486    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:03.915646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:03.915606    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:09.007683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:09.007604    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:13.496675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:13.496601    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:14.837452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:14.837413    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:15.259231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:15.256893    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:15.731226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:15.731162    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:16.231235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.228640    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:16.281139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.253701    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:16.653072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.653026    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"]
Oct 30 22:14:16.653661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.653617    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy" containerID="cri-o://a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de" gracePeriod=30
Oct 30 22:14:16.870382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.870329    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:16.921624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.921516    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:14:16.923051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.922836    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:14:16.924097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:16.924064    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Oct 30 22:14:17.127178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.127113    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:17.151237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.129148    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:17.180961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.151964    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"]
Oct 30 22:14:17.181402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.181377    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cce86f1-jj8ff"]
Oct 30 22:14:17.291239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.283297    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:14:17.291239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.284050    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:14:17.576806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.576705    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:17.935162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.935014    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:18.005460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.949710    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"]
Oct 30 22:14:18.006771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:17.970795    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:18.049012 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.048981    5031 cpu_manager.go:414] "containerMap: removing container" podUID="03fc08e7-13f0-482e-a150-1af21e3e8d23" containerName="revoke-keys"
Oct 30 22:14:18.049306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.049290    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fc08e7-13f0-482e-a150-1af21e3e8d23" containerName="revoke-keys"
Oct 30 22:14:18.049438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.049427    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" containerName="payload-processing"
Oct 30 22:14:18.049524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.049513    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" containerName="payload-processing"
Oct 30 22:14:18.049863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.049851    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fc08e7-13f0-482e-a150-1af21e3e8d23" containerName="revoke-keys"
Oct 30 22:14:18.049957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.049948    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cf1e3b-0931-44f1-b3e3-cb1f7f8f948f" containerName="payload-processing"
Oct 30 22:14:18.067427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.067390    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"]
Oct 30 22:14:18.067628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.067446    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"]
Oct 30 22:14:18.067628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.067501    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:14:18.067628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.067536    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:14:18.067628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.067563    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:14:18.068844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.068813    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.069936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.069910    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08235cc9-n2dtp"]
Oct 30 22:14:18.162235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.151644    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ef8140e-openshift-default-dockercfg-24rvz\""
Oct 30 22:14:18.176952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.176918    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.177688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.177668    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.177949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.177929    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.178139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.178104    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771f3c50-5456-4960-9921-406fc388b22e-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.178453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.178437    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.179647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.179626    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771f3c50-5456-4960-9921-406fc388b22e-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.179892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.179875    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhtp\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-kube-api-access-5nhtp\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.180198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.180180    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.180485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.180467    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287063    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287111    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287158    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287187    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771f3c50-5456-4960-9921-406fc388b22e-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287320    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287471    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771f3c50-5456-4960-9921-406fc388b22e-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287537    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhtp\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-kube-api-access-5nhtp\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287653    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.287792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.287715    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.288418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.288303    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.288602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.288579    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.289323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.289296    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.289911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.289885    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771f3c50-5456-4960-9921-406fc388b22e-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.290060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.290033    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.292940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.292912    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771f3c50-5456-4960-9921-406fc388b22e-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.293399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.293385    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.389233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.375841    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhtp\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-kube-api-access-5nhtp\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.389233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.382116    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.432080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.429089    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:18.596046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.596013    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"]
Oct 30 22:14:18.597246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.596990    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy" containerID="cri-o://ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8" gracePeriod=30
Oct 30 22:14:18.971184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.970532    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f" path="/var/lib/kubelet/pods/5ef7cdeb-bbbd-4e0d-9dde-a06b76e1743f/volumes"
Oct 30 22:14:18.971393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:18.971200    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8324b0c6-c8ab-44f9-a7f0-70eb39b1320b" path="/var/lib/kubelet/pods/8324b0c6-c8ab-44f9-a7f0-70eb39b1320b/volumes"
Oct 30 22:14:19.231053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:19.230963    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:19.244420 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:14:19.244391    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771f3c50_5456_4960_9921_406fc388b22e.slice/crio-0385b6c7625fe6aaa10acaebac852736375dce6b278939270d02e3d841aded6d WatchSource:0}: Error finding container 0385b6c7625fe6aaa10acaebac852736375dce6b278939270d02e3d841aded6d: Status 404 returned error can't find the container with id 0385b6c7625fe6aaa10acaebac852736375dce6b278939270d02e3d841aded6d
Oct 30 22:14:19.247611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:19.247574    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:19.247708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:19.247653    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:19.247708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:19.247687    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:19.250101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:19.250073    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"]
Oct 30 22:14:19.996301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:19.996257    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" event={"ID":"771f3c50-5456-4960-9921-406fc388b22e","Type":"ContainerStarted","Data":"b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf"}
Oct 30 22:14:19.996804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:19.996335    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" event={"ID":"771f3c50-5456-4960-9921-406fc388b22e","Type":"ContainerStarted","Data":"0385b6c7625fe6aaa10acaebac852736375dce6b278939270d02e3d841aded6d"}
Oct 30 22:14:20.124361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:20.124293    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podStartSLOduration=3.124275308 podStartE2EDuration="3.124275308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:20.1215642 +0000 UTC m=+1471.645706964" watchObservedRunningTime="2026-10-30 22:14:20.124275308 +0000 UTC m=+1471.648418073"
Oct 30 22:14:20.377235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:20.376868    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:20.438231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:20.438107    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:20.453676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:20.453642    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:20.562548 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:14:20.562499    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:14:20.606414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:20.606373    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:20.835562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:20.835472    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:21.007324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:21.007291    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:21.008341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:21.008310    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:14:21.538187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:21.538155    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:21.928350 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:21.928318    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:14:22.022273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.022236    5031 generic.go:356] "Generic (PLEG): container finished" podID="9aabc72e-c473-4680-9422-c32ddd053f82" containerID="a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de" exitCode=0
Oct 30 22:14:22.022790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.022327    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"
Oct 30 22:14:22.022790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.022353    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" event={"ID":"9aabc72e-c473-4680-9422-c32ddd053f82","Type":"ContainerDied","Data":"a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de"}
Oct 30 22:14:22.022790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.022425    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb" event={"ID":"9aabc72e-c473-4680-9422-c32ddd053f82","Type":"ContainerDied","Data":"825fe196af3afde9d5119661b1b421d155ba8f691c93a04b2bfdfd135a9047a0"}
Oct 30 22:14:22.022790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.022463    5031 scope.go:122] "RemoveContainer" containerID="a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de"
Oct 30 22:14:22.028766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.028645    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-kube-api-access-dmdrx\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-kube-api-access-dmdrx\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.028766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.028757    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-socket\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.029038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.028905    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-istio-token\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-istio-token\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.029038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.028948    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-envoy\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.029038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.028973    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-data\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.029234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.029043    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aabc72e-c473-4680-9422-c32ddd053f82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aabc72e-c473-4680-9422-c32ddd053f82-istio-podinfo\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.029234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.029159    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aabc72e-c473-4680-9422-c32ddd053f82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aabc72e-c473-4680-9422-c32ddd053f82-istiod-ca-cert\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.029342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.029258    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-certs\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.029342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.029312    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-credential-socket\") pod \"9aabc72e-c473-4680-9422-c32ddd053f82\" (UID: \"9aabc72e-c473-4680-9422-c32ddd053f82\") "
Oct 30 22:14:22.030338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.030313    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-credential-socket" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:22.030597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.030576    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aabc72e-c473-4680-9422-c32ddd053f82-istiod-ca-cert" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:22.030794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.030774    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-certs" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:22.030963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.030947    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-socket" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:22.031353 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.031324    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-data" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:22.031845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.031816    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-envoy" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:22.034357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.034334    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-istio-token" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:22.034471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.034375    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-kube-api-access-dmdrx" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "kube-api-access-dmdrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:22.034888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.034869    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aabc72e-c473-4680-9422-c32ddd053f82-istio-podinfo" pod "9aabc72e-c473-4680-9422-c32ddd053f82" (UID: "9aabc72e-c473-4680-9422-c32ddd053f82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:22.043734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.043708    5031 scope.go:122] "RemoveContainer" containerID="3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4"
Oct 30 22:14:22.072700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.072676    5031 scope.go:122] "RemoveContainer" containerID="a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de"
Oct 30 22:14:22.073064 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:14:22.073043    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de\": container with ID starting with a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de not found: ID does not exist" containerID="a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de"
Oct 30 22:14:22.073107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.073073    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de"} err="failed to get container status \"a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de\": rpc error: code = NotFound desc = could not find container \"a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de\": container with ID starting with a94ddca69fc301bcdb55686c2bf989f955f3e2aa9ae4b28b08b534bfb85250de not found: ID does not exist"
Oct 30 22:14:22.073107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.073092    5031 scope.go:122] "RemoveContainer" containerID="3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4"
Oct 30 22:14:22.073383 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:14:22.073363    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4\": container with ID starting with 3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4 not found: ID does not exist" containerID="3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4"
Oct 30 22:14:22.073434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.073392    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4"} err="failed to get container status \"3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4\": rpc error: code = NotFound desc = could not find container \"3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4\": container with ID starting with 3cac09015b8cf7465adb46f41ba371a68b7f98efa6c68b8ee5b8b1e95c8448f4 not found: ID does not exist"
Oct 30 22:14:22.132349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.132303    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aabc72e-c473-4680-9422-c32ddd053f82-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143031    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143358    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143397    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdrx\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-kube-api-access-dmdrx\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143414    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143428    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aabc72e-c473-4680-9422-c32ddd053f82-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143460    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143476    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aabc72e-c473-4680-9422-c32ddd053f82-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.143544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.143492    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aabc72e-c473-4680-9422-c32ddd053f82-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:22.365595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.365558    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:22.463253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.463190    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"]
Oct 30 22:14:22.471729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.471687    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb"]
Oct 30 22:14:22.621728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.621694    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:22.648355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.648318    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" path="/var/lib/kubelet/pods/9aabc72e-c473-4680-9422-c32ddd053f82/volumes"
Oct 30 22:14:22.860739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:22.860705    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:23.158581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.158546    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:23.921503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.921467    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:14:23.983020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.982985    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-credential-socket\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983063    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-certs\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983114    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-envoy\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983421    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-podinfo\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983439    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-certs" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:23.983562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983493    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-socket\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983553    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-token\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983531    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-credential-socket" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:23.983717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983656    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-data\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983717    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-kube-api-access-jjjpf\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-kube-api-access-jjjpf\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.983777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983735    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-socket" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:23.983946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.983925    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6a805d-46e7-499f-b443-fde8acc5863a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a805d-46e7-499f-b443-fde8acc5863a-istiod-ca-cert\") pod \"1f6a805d-46e7-499f-b443-fde8acc5863a\" (UID: \"1f6a805d-46e7-499f-b443-fde8acc5863a\") "
Oct 30 22:14:23.984055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.984009    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-data" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:23.984705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.984660    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6a805d-46e7-499f-b443-fde8acc5863a-istiod-ca-cert" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:23.985135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.985037    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.985276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.985250    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.985412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.985283    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.985720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.985453    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.985817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.985749    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a805d-46e7-499f-b443-fde8acc5863a-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:23.986887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.986865    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-podinfo" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:23.987058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.987033    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-token" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:23.987579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.987556    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-envoy" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:23.988118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:23.988095    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-kube-api-access-jjjpf" pod "1f6a805d-46e7-499f-b443-fde8acc5863a" (UID: "1f6a805d-46e7-499f-b443-fde8acc5863a"). InnerVolumeSpecName "kube-api-access-jjjpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.126927    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.126963    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.126978    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.126993    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjpf\" (UniqueName: \"kubernetes.io/projected/1f6a805d-46e7-499f-b443-fde8acc5863a-kube-api-access-jjjpf\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.129406    5031 generic.go:356] "Generic (PLEG): container finished" podID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerID="ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8" exitCode=0
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.129815    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" event={"ID":"1f6a805d-46e7-499f-b443-fde8acc5863a","Type":"ContainerDied","Data":"ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8"}
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.129895    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt" event={"ID":"1f6a805d-46e7-499f-b443-fde8acc5863a","Type":"ContainerDied","Data":"1866cccbcb32bb5d355031aa1b96b6abd7f9dc53953076c49484bb27ee6f59ff"}
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.129922    5031 scope.go:122] "RemoveContainer" containerID="ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8"
Oct 30 22:14:24.137467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.130083    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"
Oct 30 22:14:24.193294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.193269    5031 scope.go:122] "RemoveContainer" containerID="df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325"
Oct 30 22:14:24.218988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.218646    5031 scope.go:122] "RemoveContainer" containerID="ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8"
Oct 30 22:14:24.220068 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:14:24.219729    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8\": container with ID starting with ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8 not found: ID does not exist" containerID="ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8"
Oct 30 22:14:24.220068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.219783    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8"} err="failed to get container status \"ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8\": rpc error: code = NotFound desc = could not find container \"ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8\": container with ID starting with ff168985609390e8d7a811350e86603898a9af42d11670a729ed616ac3b226a8 not found: ID does not exist"
Oct 30 22:14:24.220068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.219813    5031 scope.go:122] "RemoveContainer" containerID="df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325"
Oct 30 22:14:24.220488 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:14:24.220404    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325\": container with ID starting with df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325 not found: ID does not exist" containerID="df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325"
Oct 30 22:14:24.220488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.220437    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325"} err="failed to get container status \"df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325\": rpc error: code = NotFound desc = could not find container \"df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325\": container with ID starting with df3a6e39e28ecc38e3693c266ff3a507e9ed8fb90133cbac27dedfaced8c9325 not found: ID does not exist"
Oct 30 22:14:24.334057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.333974    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"]
Oct 30 22:14:24.344415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.344367    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt"]
Oct 30 22:14:24.650488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:24.650452    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" path="/var/lib/kubelet/pods/1f6a805d-46e7-499f-b443-fde8acc5863a/volumes"
Oct 30 22:14:25.379848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:25.379810    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:25.646327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:25.646231    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:25.930305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:25.926461    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:26.214478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.214386    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"]
Oct 30 22:14:26.225533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.225499    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:26.260716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.260683    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643a82de-d4sff"]
Oct 30 22:14:26.380136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.380096    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"]
Oct 30 22:14:26.380985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.380965    5031 cpu_manager.go:414] "containerMap: removing container" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy"
Oct 30 22:14:26.381048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.380987    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy"
Oct 30 22:14:26.381048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381003    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy"
Oct 30 22:14:26.381048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381009    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy"
Oct 30 22:14:26.381048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381027    5031 cpu_manager.go:414] "containerMap: removing container" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy"
Oct 30 22:14:26.381048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381033    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy"
Oct 30 22:14:26.381257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381141    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy"
Oct 30 22:14:26.381257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381149    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy"
Oct 30 22:14:26.381257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381159    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy"
Oct 30 22:14:26.381257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.381172    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aabc72e-c473-4680-9422-c32ddd053f82" containerName="istio-proxy"
Oct 30 22:14:26.385346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.385324    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.404091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.404060    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995rs\" (UniqueName: \"kubernetes.io/projected/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-kube-api-access-995rs\") pod \"payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") " pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.404438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.404411    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") " pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.453237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.452585    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3ef8140e-dockercfg-pvddp\""
Oct 30 22:14:26.453456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.453331    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3ef8140e\""
Oct 30 22:14:26.552072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.549703    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995rs\" (UniqueName: \"kubernetes.io/projected/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-kube-api-access-995rs\") pod \"payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") " pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.552072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.549943    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") " pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.552072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.550956    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") " pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.593297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.579291    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"]
Oct 30 22:14:26.646576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.645399    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:26.650289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.648378    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-995rs\" (UniqueName: \"kubernetes.io/projected/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-kube-api-access-995rs\") pod \"payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") " pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.658972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.658931    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03fc08e7-13f0-482e-a150-1af21e3e8d23" path="/var/lib/kubelet/pods/03fc08e7-13f0-482e-a150-1af21e3e8d23/volumes"
Oct 30 22:14:26.659493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.659471    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"]
Oct 30 22:14:26.660859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.660833    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy"
Oct 30 22:14:26.660859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.660857    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6a805d-46e7-499f-b443-fde8acc5863a" containerName="istio-proxy"
Oct 30 22:14:26.670917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.670887    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"]
Oct 30 22:14:26.671062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.670997    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:26.701609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.699788    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:26.764581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.754648    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:26.764581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.754741    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7ts\" (UniqueName: \"kubernetes.io/projected/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-kube-api-access-mw7ts\") pod \"payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:26.857521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.856997    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:26.857521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.857069    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7ts\" (UniqueName: \"kubernetes.io/projected/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-kube-api-access-mw7ts\") pod \"payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:26.857896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.857870    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:26.902170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.902135    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:26.933259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:26.931532    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7ts\" (UniqueName: \"kubernetes.io/projected/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-kube-api-access-mw7ts\") pod \"payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:27.004445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:27.004405    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:27.158550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:27.158507    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:27.213015 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:14:27.212981    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b88a3fc_bcaf_46d2_aeee_460ec6df3ed0.slice/crio-03a12d8347f530f7d5e0510be2f01cc04edcb24d1bb2782fd4f120b2c3b841e5 WatchSource:0}: Error finding container 03a12d8347f530f7d5e0510be2f01cc04edcb24d1bb2782fd4f120b2c3b841e5: Status 404 returned error can't find the container with id 03a12d8347f530f7d5e0510be2f01cc04edcb24d1bb2782fd4f120b2c3b841e5
Oct 30 22:14:27.216825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:27.216800    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"]
Oct 30 22:14:27.398584 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:14:27.398554    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7ddab7_0613_4b52_9ea6_7f2148d64f11.slice/crio-2c443ccaca5fa37ab9a6c19dc3699487f1b3e458c52d24f332c4c98e8616a27d WatchSource:0}: Error finding container 2c443ccaca5fa37ab9a6c19dc3699487f1b3e458c52d24f332c4c98e8616a27d: Status 404 returned error can't find the container with id 2c443ccaca5fa37ab9a6c19dc3699487f1b3e458c52d24f332c4c98e8616a27d
Oct 30 22:14:27.401493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:27.401464    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"]
Oct 30 22:14:28.183480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.183432    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv" event={"ID":"ba7ddab7-0613-4b52-9ea6-7f2148d64f11","Type":"ContainerStarted","Data":"cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173"}
Oct 30 22:14:28.183682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.183498    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv" event={"ID":"ba7ddab7-0613-4b52-9ea6-7f2148d64f11","Type":"ContainerStarted","Data":"2c443ccaca5fa37ab9a6c19dc3699487f1b3e458c52d24f332c4c98e8616a27d"}
Oct 30 22:14:28.183682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.183528    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:28.185484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.185447    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5" event={"ID":"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0","Type":"ContainerStarted","Data":"abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f"}
Oct 30 22:14:28.185631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.185509    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5" event={"ID":"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0","Type":"ContainerStarted","Data":"03a12d8347f530f7d5e0510be2f01cc04edcb24d1bb2782fd4f120b2c3b841e5"}
Oct 30 22:14:28.185631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.185546    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:28.306335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.306286    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv" podStartSLOduration=2.306271493 podStartE2EDuration="2.306271493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:28.30349577 +0000 UTC m=+1479.827638554" watchObservedRunningTime="2026-10-30 22:14:28.306271493 +0000 UTC m=+1479.830414256"
Oct 30 22:14:28.434605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:28.433347    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5" podStartSLOduration=2.433326761 podStartE2EDuration="2.433326761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:28.42384014 +0000 UTC m=+1479.947982906" watchObservedRunningTime="2026-10-30 22:14:28.433326761 +0000 UTC m=+1479.957469526"
Oct 30 22:14:30.020458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.020419    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:30.251387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.251348    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:30.577233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.571872    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"]
Oct 30 22:14:30.634728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.624344    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:30.703802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.703767    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-781ccf27-dockercfg-g6x6k\""
Oct 30 22:14:30.727520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.707052    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fnl\" (UniqueName: \"kubernetes.io/projected/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-kube-api-access-h6fnl\") pod \"payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") " pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:30.810035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.809999    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-plugins-config-volume\") pod \"payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") " pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:30.810582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.755440    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-781ccf27\""
Oct 30 22:14:30.844745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.844714    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"]
Oct 30 22:14:30.912600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.912560    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-plugins-config-volume\") pod \"payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") " pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:30.912917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.912888    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fnl\" (UniqueName: \"kubernetes.io/projected/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-kube-api-access-h6fnl\") pod \"payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") " pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:30.913525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.913487    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-plugins-config-volume\") pod \"payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") " pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:30.976661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:30.976631    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fnl\" (UniqueName: \"kubernetes.io/projected/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-kube-api-access-h6fnl\") pod \"payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") " pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:31.112656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:31.112568    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:31.508098 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:14:31.508067    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c2d1ac_49b7_41b0_b34d_98bb5c0d0597.slice/crio-55d2e8dedbbdba6aeee831dd1ab022211273008486ff0b4281b1db93ec12e279 WatchSource:0}: Error finding container 55d2e8dedbbdba6aeee831dd1ab022211273008486ff0b4281b1db93ec12e279: Status 404 returned error can't find the container with id 55d2e8dedbbdba6aeee831dd1ab022211273008486ff0b4281b1db93ec12e279
Oct 30 22:14:31.510467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:31.510445    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"]
Oct 30 22:14:31.745320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:31.743244    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:32.237988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:32.237952    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6" event={"ID":"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597","Type":"ContainerStarted","Data":"1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f"}
Oct 30 22:14:32.238534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:32.238011    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6" event={"ID":"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597","Type":"ContainerStarted","Data":"55d2e8dedbbdba6aeee831dd1ab022211273008486ff0b4281b1db93ec12e279"}
Oct 30 22:14:32.238534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:32.238067    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:32.366283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:32.360134    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6" podStartSLOduration=2.360114425 podStartE2EDuration="2.360114425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:32.358869385 +0000 UTC m=+1483.883012507" watchObservedRunningTime="2026-10-30 22:14:32.360114425 +0000 UTC m=+1483.884257191"
Oct 30 22:14:37.330546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:37.330503    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:39.066827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:39.066785    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"]
Oct 30 22:14:39.117241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:39.117191    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba3954-bls6x"]
Oct 30 22:14:39.198852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:39.198821    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:14:39.199058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:39.198897    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:14:40.116534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:40.116497    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"]
Oct 30 22:14:40.167062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:40.167026    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f5a8ba-2hnw5"]
Oct 30 22:14:40.651479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:40.651444    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55fd6fd0-f5e8-4c24-855a-25dab5bc964f" path="/var/lib/kubelet/pods/55fd6fd0-f5e8-4c24-855a-25dab5bc964f/volumes"
Oct 30 22:14:40.652138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:40.652114    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba24195f-e4a5-44dc-993c-bc5de84df48c" path="/var/lib/kubelet/pods/ba24195f-e4a5-44dc-993c-bc5de84df48c/volumes"
Oct 30 22:14:42.395849 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:42.395808    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:43.253405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:43.253370    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:14:47.293165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.293129    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:47.528933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.528891    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:47.753516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.753482    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:47.879021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.878983    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"]
Oct 30 22:14:47.883952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.883927    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:47.938402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.938372    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:14:47.938402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.938372    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:14:47.938666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.938564    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:14:47.988123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.988092    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6s5w5\""
Oct 30 22:14:47.997965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.997932    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"]
Oct 30 22:14:47.999671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:47.999642    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:48.028520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.028434    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvvz\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-kube-api-access-qcvvz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.028883 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.028851    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.029569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.029525    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9ad58e-6de2-42c1-858a-35673ce832a0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.029824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.029803    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.030033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.030013    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.030135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.030052    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.030135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.030081    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.131572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.131536    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9ad58e-6de2-42c1-858a-35673ce832a0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.131777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.131664    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.131860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.131775    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.131860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.131807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.131860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.131832    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.132025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.131940    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvvz\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-kube-api-access-qcvvz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.132098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.132026    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.133196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.133167    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.134625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.134590    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9ad58e-6de2-42c1-858a-35673ce832a0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.135093 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.135064    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.135189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.135100    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.135301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.135186    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.195097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.195063    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.195315 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.195286    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvvz\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-kube-api-access-qcvvz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6dg8m\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.198143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.198107    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:48.647653 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:14:48.647621    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9ad58e_6de2_42c1_858a_35673ce832a0.slice/crio-14169c64d254e83941de92011d6a84e617f7279481b53e080b7775b9a04726ba WatchSource:0}: Error finding container 14169c64d254e83941de92011d6a84e617f7279481b53e080b7775b9a04726ba: Status 404 returned error can't find the container with id 14169c64d254e83941de92011d6a84e617f7279481b53e080b7775b9a04726ba
Oct 30 22:14:48.653708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:48.653684    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"]
Oct 30 22:14:49.467324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:49.467282    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m" event={"ID":"7b9ad58e-6de2-42c1-858a-35673ce832a0","Type":"ContainerStarted","Data":"14169c64d254e83941de92011d6a84e617f7279481b53e080b7775b9a04726ba"}
Oct 30 22:14:49.565329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:49.565300    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:14:49.569187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:49.569161    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:14:50.061951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:50.061903    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"]
Oct 30 22:14:50.112126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:50.112081    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"]
Oct 30 22:14:50.118259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:50.118224    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa8f78-pdjm9"]
Oct 30 22:14:50.124690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:50.124657    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f1e9b7-zsgwb"]
Oct 30 22:14:50.559489 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:14:50.559445    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:14:50.654456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:50.654416    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4b885a-a7c9-4241-9962-74330b443379" path="/var/lib/kubelet/pods/9e4b885a-a7c9-4241-9962-74330b443379/volumes"
Oct 30 22:14:50.655162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:50.655142    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb42d627-3e1b-4e36-80f4-a9402d191662" path="/var/lib/kubelet/pods/eb42d627-3e1b-4e36-80f4-a9402d191662/volumes"
Oct 30 22:14:51.459280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:51.457375    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:14:52.462600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:52.462566    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:53.246174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:53.246131    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:53.246315 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:53.246242    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:14:53.554893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:53.554794    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m" event={"ID":"7b9ad58e-6de2-42c1-858a-35673ce832a0","Type":"ContainerStarted","Data":"d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04"}
Oct 30 22:14:53.554893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:53.554881    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:53.791557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:53.791488    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m" podStartSLOduration=2.198026811 podStartE2EDuration="6.791468503s" totalImagesPullingTime="4.593441692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:53.736964705 +0000 UTC m=+1505.261107502" watchObservedRunningTime="2026-10-30 22:14:53.791468503 +0000 UTC m=+1505.315611268"
Oct 30 22:14:54.573674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:54.573633    5031 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:54.574104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:54.573713    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m" podUID="7b9ad58e-6de2-42c1-858a-35673ce832a0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:57.338367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:57.338329    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:57.572033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:57.572001    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:14:57.733119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:14:57.733081    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.193194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.193159    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"]
Oct 30 22:15:00.197502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.197479    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.199735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.199708    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"]
Oct 30 22:15:00.203973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.203948    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.206170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.206148    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"]
Oct 30 22:15:00.209736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.209719    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.243528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.243501    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"]
Oct 30 22:15:00.246896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.246880    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.260232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.260189    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"]
Oct 30 22:15:00.260811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.260786    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:00.263109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.263093    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:00.263762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.263742    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.287897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.287879    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:15:00.288697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.288678    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6810b1b3-158a-4c13-b34d-bd87de0806d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.288778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.288741    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrlz\" (UniqueName: \"kubernetes.io/projected/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-kube-api-access-gxrlz\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.288939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.288922    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.289013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.288992    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kn8\" (UniqueName: \"kubernetes.io/projected/a0d8db54-4028-4595-8bf7-e44edacf5c20-kube-api-access-r9kn8\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.289071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.289045    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjkx\" (UniqueName: \"kubernetes.io/projected/6810b1b3-158a-4c13-b34d-bd87de0806d6-kube-api-access-4xjkx\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.289182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.289139    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fe10e0-ecfd-458e-bf13-6343539b47f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.289182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.289170    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzj8\" (UniqueName: \"kubernetes.io/projected/40fe10e0-ecfd-458e-bf13-6343539b47f7-kube-api-access-7bzj8\") pod \"maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.289310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.289189    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8db54-4028-4595-8bf7-e44edacf5c20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.313919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.313853    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"]
Oct 30 22:15:00.320937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.320905    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"]
Oct 30 22:15:00.328764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.328736    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"]
Oct 30 22:15:00.341554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.341525    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"]
Oct 30 22:15:00.388691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.387892    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"]
Oct 30 22:15:00.396579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.396550    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrlz\" (UniqueName: \"kubernetes.io/projected/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-kube-api-access-gxrlz\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.396765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.396699    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.396845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.396766    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kn8\" (UniqueName: \"kubernetes.io/projected/a0d8db54-4028-4595-8bf7-e44edacf5c20-kube-api-access-r9kn8\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.396845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.396813    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjkx\" (UniqueName: \"kubernetes.io/projected/6810b1b3-158a-4c13-b34d-bd87de0806d6-kube-api-access-4xjkx\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.396948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.396863    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fe10e0-ecfd-458e-bf13-6343539b47f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.397010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.396956    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzj8\" (UniqueName: \"kubernetes.io/projected/40fe10e0-ecfd-458e-bf13-6343539b47f7-kube-api-access-7bzj8\") pod \"maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.397010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.396985    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8db54-4028-4595-8bf7-e44edacf5c20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.397193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.397172    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6pl\" (UniqueName: \"kubernetes.io/projected/6879a500-443c-4f63-8b36-89e62aa542ed-kube-api-access-js6pl\") pod \"maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.397365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.397229    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6810b1b3-158a-4c13-b34d-bd87de0806d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.397365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.397261    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6879a500-443c-4f63-8b36-89e62aa542ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.398155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.398123    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.398396 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.398377    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fe10e0-ecfd-458e-bf13-6343539b47f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.398870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.398846    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8db54-4028-4595-8bf7-e44edacf5c20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.399019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.398996    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6810b1b3-158a-4c13-b34d-bd87de0806d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.461824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.461740    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrlz\" (UniqueName: \"kubernetes.io/projected/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-kube-api-access-gxrlz\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.462660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.462638    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kn8\" (UniqueName: \"kubernetes.io/projected/a0d8db54-4028-4595-8bf7-e44edacf5c20-kube-api-access-r9kn8\") pod \"maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.463142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.463126    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjkx\" (UniqueName: \"kubernetes.io/projected/6810b1b3-158a-4c13-b34d-bd87de0806d6-kube-api-access-4xjkx\") pod \"maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.464403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.464376    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzj8\" (UniqueName: \"kubernetes.io/projected/40fe10e0-ecfd-458e-bf13-6343539b47f7-kube-api-access-7bzj8\") pod \"maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.499321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.499288    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6pl\" (UniqueName: \"kubernetes.io/projected/6879a500-443c-4f63-8b36-89e62aa542ed-kube-api-access-js6pl\") pod \"maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.499488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.499330    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6879a500-443c-4f63-8b36-89e62aa542ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.499987 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.499967    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6879a500-443c-4f63-8b36-89e62aa542ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.515691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.515660    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:00.525027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.525004    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:00.532944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.532919    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:00.561616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.561519    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:00.563923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.563894    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6pl\" (UniqueName: \"kubernetes.io/projected/6879a500-443c-4f63-8b36-89e62aa542ed-kube-api-access-js6pl\") pod \"maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.582880 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.577865    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:00.931031 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:15:00.931002    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fe10e0_ecfd_458e_bf13_6343539b47f7.slice/crio-21450ee31da9cd43accc6e2ddb1236dabea7ac5300b92aa894b91cbf18b97e18 WatchSource:0}: Error finding container 21450ee31da9cd43accc6e2ddb1236dabea7ac5300b92aa894b91cbf18b97e18: Status 404 returned error can't find the container with id 21450ee31da9cd43accc6e2ddb1236dabea7ac5300b92aa894b91cbf18b97e18
Oct 30 22:15:00.934149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.934119    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"]
Oct 30 22:15:00.981293 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:15:00.981187    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d8db54_4028_4595_8bf7_e44edacf5c20.slice/crio-f23fda1ca5b77dd9bbc5d906d5cf4f920416c160dec7d4e96b0ac801bcc7306f WatchSource:0}: Error finding container f23fda1ca5b77dd9bbc5d906d5cf4f920416c160dec7d4e96b0ac801bcc7306f: Status 404 returned error can't find the container with id f23fda1ca5b77dd9bbc5d906d5cf4f920416c160dec7d4e96b0ac801bcc7306f
Oct 30 22:15:00.983072 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:15:00.983049    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4092e05f_8cc3_45c6_87a4_4d092b5e0b6e.slice/crio-0acc95979054db534e5071a9ba83698fa42382e0bc78c766147f50c041ba9eb5 WatchSource:0}: Error finding container 0acc95979054db534e5071a9ba83698fa42382e0bc78c766147f50c041ba9eb5: Status 404 returned error can't find the container with id 0acc95979054db534e5071a9ba83698fa42382e0bc78c766147f50c041ba9eb5
Oct 30 22:15:00.983900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.983881    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"]
Oct 30 22:15:00.990874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:00.990851    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"]
Oct 30 22:15:01.038893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.038869    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"]
Oct 30 22:15:01.058675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.058648    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"]
Oct 30 22:15:01.110328 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:15:01.110294    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6810b1b3_158a_4c13_b34d_bd87de0806d6.slice/crio-45ee87298942e4312f4d78ee91d93ec95ff3fa839f06247fdb1ace2ac9a81573 WatchSource:0}: Error finding container 45ee87298942e4312f4d78ee91d93ec95ff3fa839f06247fdb1ace2ac9a81573: Status 404 returned error can't find the container with id 45ee87298942e4312f4d78ee91d93ec95ff3fa839f06247fdb1ace2ac9a81573
Oct 30 22:15:01.111868 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:15:01.111843    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6879a500_443c_4f63_8b36_89e62aa542ed.slice/crio-7f53a46cfbf82b686897440c712f24271cbd80a2ebabcb052f4038a7d5efcaa1 WatchSource:0}: Error finding container 7f53a46cfbf82b686897440c712f24271cbd80a2ebabcb052f4038a7d5efcaa1: Status 404 returned error can't find the container with id 7f53a46cfbf82b686897440c712f24271cbd80a2ebabcb052f4038a7d5efcaa1
Oct 30 22:15:01.704051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.704014    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" event={"ID":"a0d8db54-4028-4595-8bf7-e44edacf5c20","Type":"ContainerStarted","Data":"6b9c555f87a74ce06ffbfb46dc4e7aca8a3b6cad7436c9fb77676ba62ed8829e"}
Oct 30 22:15:01.704517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.704073    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" event={"ID":"a0d8db54-4028-4595-8bf7-e44edacf5c20","Type":"ContainerStarted","Data":"f23fda1ca5b77dd9bbc5d906d5cf4f920416c160dec7d4e96b0ac801bcc7306f"}
Oct 30 22:15:01.705971 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.705945    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" event={"ID":"40fe10e0-ecfd-458e-bf13-6343539b47f7","Type":"ContainerStarted","Data":"b93ca594b97f9105dacfaecacb215cde8dbd3ef8a8291a768ed1a64adf90aef1"}
Oct 30 22:15:01.706141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.705994    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" event={"ID":"40fe10e0-ecfd-458e-bf13-6343539b47f7","Type":"ContainerStarted","Data":"21450ee31da9cd43accc6e2ddb1236dabea7ac5300b92aa894b91cbf18b97e18"}
Oct 30 22:15:01.707477 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.707449    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" event={"ID":"6810b1b3-158a-4c13-b34d-bd87de0806d6","Type":"ContainerStarted","Data":"45ee87298942e4312f4d78ee91d93ec95ff3fa839f06247fdb1ace2ac9a81573"}
Oct 30 22:15:01.709641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.709615    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" event={"ID":"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e","Type":"ContainerStarted","Data":"6f4fd61830b6fa678ff4eb33e01b452ed932e5274fbfb90b858f6de0dc3e3551"}
Oct 30 22:15:01.709784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.709664    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" event={"ID":"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e","Type":"ContainerStarted","Data":"0acc95979054db534e5071a9ba83698fa42382e0bc78c766147f50c041ba9eb5"}
Oct 30 22:15:01.711313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.711277    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" event={"ID":"6879a500-443c-4f63-8b36-89e62aa542ed","Type":"ContainerStarted","Data":"7f53a46cfbf82b686897440c712f24271cbd80a2ebabcb052f4038a7d5efcaa1"}
Oct 30 22:15:01.844620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.844556    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" podStartSLOduration=1.844540716 podStartE2EDuration="1.844540716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.83860347 +0000 UTC m=+1513.362746253" watchObservedRunningTime="2026-10-30 22:15:01.844540716 +0000 UTC m=+1513.368683477"
Oct 30 22:15:01.964509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:01.964448    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" podStartSLOduration=1.964430477 podStartE2EDuration="1.964430477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.961098321 +0000 UTC m=+1513.485241084" watchObservedRunningTime="2026-10-30 22:15:01.964430477 +0000 UTC m=+1513.488573242"
Oct 30 22:15:02.096817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.096753    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" podStartSLOduration=2.096735755 podStartE2EDuration="2.096735755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.093534351 +0000 UTC m=+1513.617677116" watchObservedRunningTime="2026-10-30 22:15:02.096735755 +0000 UTC m=+1513.620878518"
Oct 30 22:15:02.736109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.736005    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerID="6b9c555f87a74ce06ffbfb46dc4e7aca8a3b6cad7436c9fb77676ba62ed8829e" exitCode=60
Oct 30 22:15:02.736581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.736084    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" event={"ID":"a0d8db54-4028-4595-8bf7-e44edacf5c20","Type":"ContainerDied","Data":"6b9c555f87a74ce06ffbfb46dc4e7aca8a3b6cad7436c9fb77676ba62ed8829e"}
Oct 30 22:15:02.736581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.736486    5031 scope.go:122] "RemoveContainer" containerID="6b9c555f87a74ce06ffbfb46dc4e7aca8a3b6cad7436c9fb77676ba62ed8829e"
Oct 30 22:15:02.738637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.738580    5031 generic.go:356] "Generic (PLEG): container finished" podID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerID="b93ca594b97f9105dacfaecacb215cde8dbd3ef8a8291a768ed1a64adf90aef1" exitCode=60
Oct 30 22:15:02.738728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.738651    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" event={"ID":"40fe10e0-ecfd-458e-bf13-6343539b47f7","Type":"ContainerDied","Data":"b93ca594b97f9105dacfaecacb215cde8dbd3ef8a8291a768ed1a64adf90aef1"}
Oct 30 22:15:02.739086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.739069    5031 scope.go:122] "RemoveContainer" containerID="b93ca594b97f9105dacfaecacb215cde8dbd3ef8a8291a768ed1a64adf90aef1"
Oct 30 22:15:02.741088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.741059    5031 generic.go:356] "Generic (PLEG): container finished" podID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerID="f5dc772073b8e678bcc99a6809201d39d8223126767d257f979ea98bd9c247e8" exitCode=60
Oct 30 22:15:02.741230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.741089    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" event={"ID":"6810b1b3-158a-4c13-b34d-bd87de0806d6","Type":"ContainerDied","Data":"f5dc772073b8e678bcc99a6809201d39d8223126767d257f979ea98bd9c247e8"}
Oct 30 22:15:02.741761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.741448    5031 scope.go:122] "RemoveContainer" containerID="f5dc772073b8e678bcc99a6809201d39d8223126767d257f979ea98bd9c247e8"
Oct 30 22:15:02.743754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.743734    5031 generic.go:356] "Generic (PLEG): container finished" podID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerID="6f4fd61830b6fa678ff4eb33e01b452ed932e5274fbfb90b858f6de0dc3e3551" exitCode=60
Oct 30 22:15:02.743878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.743826    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" event={"ID":"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e","Type":"ContainerDied","Data":"6f4fd61830b6fa678ff4eb33e01b452ed932e5274fbfb90b858f6de0dc3e3551"}
Oct 30 22:15:02.744057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.744043    5031 scope.go:122] "RemoveContainer" containerID="6f4fd61830b6fa678ff4eb33e01b452ed932e5274fbfb90b858f6de0dc3e3551"
Oct 30 22:15:02.752956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.752905    5031 generic.go:356] "Generic (PLEG): container finished" podID="6879a500-443c-4f63-8b36-89e62aa542ed" containerID="ffa9c35c0b34ca067ae1a858413610fb106b9ab9d3018276ea31c53d48a2f440" exitCode=60
Oct 30 22:15:02.753089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.752971    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" event={"ID":"6879a500-443c-4f63-8b36-89e62aa542ed","Type":"ContainerDied","Data":"ffa9c35c0b34ca067ae1a858413610fb106b9ab9d3018276ea31c53d48a2f440"}
Oct 30 22:15:02.753327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.753301    5031 scope.go:122] "RemoveContainer" containerID="ffa9c35c0b34ca067ae1a858413610fb106b9ab9d3018276ea31c53d48a2f440"
Oct 30 22:15:02.995568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:02.995435    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:03.766202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:03.766165    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" event={"ID":"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e","Type":"ContainerStarted","Data":"bdccbbc8cadbbef018758ad66e4461869f3b2135b2ff4068c4abb3b567e01b26"}
Oct 30 22:15:03.768458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:03.768428    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" event={"ID":"6879a500-443c-4f63-8b36-89e62aa542ed","Type":"ContainerStarted","Data":"42edd9a5ddafeed2f3e3330316bf9e1aed78d830029c23219a42abcfee7d01ec"}
Oct 30 22:15:03.770498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:03.770466    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" event={"ID":"a0d8db54-4028-4595-8bf7-e44edacf5c20","Type":"ContainerStarted","Data":"8f7ece2e0989153bb99018dc239a4aeaaa2665b816b66d2aa223e3f107d18dac"}
Oct 30 22:15:03.772690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:03.772663    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" event={"ID":"40fe10e0-ecfd-458e-bf13-6343539b47f7","Type":"ContainerStarted","Data":"32888afb041d8dce6c1a65bdab8020f3c05378ea87803871f7c50f935649e090"}
Oct 30 22:15:03.774670 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:03.774647    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" event={"ID":"6810b1b3-158a-4c13-b34d-bd87de0806d6","Type":"ContainerStarted","Data":"31e5ec4b36f7218b89339161e082553c21c1e3fdbd44701ee43b81bd06dcbf41"}
Oct 30 22:15:04.130179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.130121    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" podStartSLOduration=4.130103889 podStartE2EDuration="4.130103889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.126281758 +0000 UTC m=+1515.650424523" watchObservedRunningTime="2026-10-30 22:15:04.130103889 +0000 UTC m=+1515.654246655"
Oct 30 22:15:04.254515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.254442    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" podStartSLOduration=4.254421703 podStartE2EDuration="4.254421703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.246517554 +0000 UTC m=+1515.770660518" watchObservedRunningTime="2026-10-30 22:15:04.254421703 +0000 UTC m=+1515.778564469"
Oct 30 22:15:04.786863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.786822    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerID="8f7ece2e0989153bb99018dc239a4aeaaa2665b816b66d2aa223e3f107d18dac" exitCode=60
Oct 30 22:15:04.787373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.786903    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" event={"ID":"a0d8db54-4028-4595-8bf7-e44edacf5c20","Type":"ContainerDied","Data":"8f7ece2e0989153bb99018dc239a4aeaaa2665b816b66d2aa223e3f107d18dac"}
Oct 30 22:15:04.787373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.786972    5031 scope.go:122] "RemoveContainer" containerID="6b9c555f87a74ce06ffbfb46dc4e7aca8a3b6cad7436c9fb77676ba62ed8829e"
Oct 30 22:15:04.787373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.787272    5031 scope.go:122] "RemoveContainer" containerID="8f7ece2e0989153bb99018dc239a4aeaaa2665b816b66d2aa223e3f107d18dac"
Oct 30 22:15:04.787640 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:04.787616    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c_odh-ai-gateway-infra(a0d8db54-4028-4595-8bf7-e44edacf5c20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20"
Oct 30 22:15:04.789246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.789192    5031 generic.go:356] "Generic (PLEG): container finished" podID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerID="32888afb041d8dce6c1a65bdab8020f3c05378ea87803871f7c50f935649e090" exitCode=60
Oct 30 22:15:04.789370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.789245    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" event={"ID":"40fe10e0-ecfd-458e-bf13-6343539b47f7","Type":"ContainerDied","Data":"32888afb041d8dce6c1a65bdab8020f3c05378ea87803871f7c50f935649e090"}
Oct 30 22:15:04.789559 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.789544    5031 scope.go:122] "RemoveContainer" containerID="32888afb041d8dce6c1a65bdab8020f3c05378ea87803871f7c50f935649e090"
Oct 30 22:15:04.789892 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:04.789864    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27_odh-ai-gateway-infra(40fe10e0-ecfd-458e-bf13-6343539b47f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7"
Oct 30 22:15:04.791744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.791723    5031 generic.go:356] "Generic (PLEG): container finished" podID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerID="31e5ec4b36f7218b89339161e082553c21c1e3fdbd44701ee43b81bd06dcbf41" exitCode=60
Oct 30 22:15:04.791867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.791852    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" event={"ID":"6810b1b3-158a-4c13-b34d-bd87de0806d6","Type":"ContainerDied","Data":"31e5ec4b36f7218b89339161e082553c21c1e3fdbd44701ee43b81bd06dcbf41"}
Oct 30 22:15:04.792082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.792065    5031 scope.go:122] "RemoveContainer" containerID="31e5ec4b36f7218b89339161e082553c21c1e3fdbd44701ee43b81bd06dcbf41"
Oct 30 22:15:04.792370 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:04.792347    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8_odh-ai-gateway-infra(6810b1b3-158a-4c13-b34d-bd87de0806d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6"
Oct 30 22:15:04.794038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.794018    5031 generic.go:356] "Generic (PLEG): container finished" podID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerID="bdccbbc8cadbbef018758ad66e4461869f3b2135b2ff4068c4abb3b567e01b26" exitCode=60
Oct 30 22:15:04.794178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.794147    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" event={"ID":"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e","Type":"ContainerDied","Data":"bdccbbc8cadbbef018758ad66e4461869f3b2135b2ff4068c4abb3b567e01b26"}
Oct 30 22:15:04.794372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.794358    5031 scope.go:122] "RemoveContainer" containerID="bdccbbc8cadbbef018758ad66e4461869f3b2135b2ff4068c4abb3b567e01b26"
Oct 30 22:15:04.794682 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:04.794658    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf_odh-ai-gateway-infra(4092e05f-8cc3-45c6-87a4-4d092b5e0b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e"
Oct 30 22:15:04.796354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.796332    5031 generic.go:356] "Generic (PLEG): container finished" podID="6879a500-443c-4f63-8b36-89e62aa542ed" containerID="42edd9a5ddafeed2f3e3330316bf9e1aed78d830029c23219a42abcfee7d01ec" exitCode=60
Oct 30 22:15:04.796459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.796378    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" event={"ID":"6879a500-443c-4f63-8b36-89e62aa542ed","Type":"ContainerDied","Data":"42edd9a5ddafeed2f3e3330316bf9e1aed78d830029c23219a42abcfee7d01ec"}
Oct 30 22:15:04.796662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.796644    5031 scope.go:122] "RemoveContainer" containerID="42edd9a5ddafeed2f3e3330316bf9e1aed78d830029c23219a42abcfee7d01ec"
Oct 30 22:15:04.796919 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:04.796901    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz_odh-ai-gateway-infra(6879a500-443c-4f63-8b36-89e62aa542ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" podUID="6879a500-443c-4f63-8b36-89e62aa542ed"
Oct 30 22:15:04.806085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.806059    5031 scope.go:122] "RemoveContainer" containerID="b93ca594b97f9105dacfaecacb215cde8dbd3ef8a8291a768ed1a64adf90aef1"
Oct 30 22:15:04.822026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.822006    5031 scope.go:122] "RemoveContainer" containerID="f5dc772073b8e678bcc99a6809201d39d8223126767d257f979ea98bd9c247e8"
Oct 30 22:15:04.838117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.838100    5031 scope.go:122] "RemoveContainer" containerID="6f4fd61830b6fa678ff4eb33e01b452ed932e5274fbfb90b858f6de0dc3e3551"
Oct 30 22:15:04.854357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:04.854335    5031 scope.go:122] "RemoveContainer" containerID="ffa9c35c0b34ca067ae1a858413610fb106b9ab9d3018276ea31c53d48a2f440"
Oct 30 22:15:05.302253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:05.302192    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:05.550373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:05.550337    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:05.809148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:05.809116    5031 scope.go:122] "RemoveContainer" containerID="bdccbbc8cadbbef018758ad66e4461869f3b2135b2ff4068c4abb3b567e01b26"
Oct 30 22:15:05.809716 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:05.809591    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf_odh-ai-gateway-infra(4092e05f-8cc3-45c6-87a4-4d092b5e0b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e"
Oct 30 22:15:05.811400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:05.811371    5031 scope.go:122] "RemoveContainer" containerID="42edd9a5ddafeed2f3e3330316bf9e1aed78d830029c23219a42abcfee7d01ec"
Oct 30 22:15:05.811824 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:05.811788    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz_odh-ai-gateway-infra(6879a500-443c-4f63-8b36-89e62aa542ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" podUID="6879a500-443c-4f63-8b36-89e62aa542ed"
Oct 30 22:15:05.813335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:05.813313    5031 scope.go:122] "RemoveContainer" containerID="8f7ece2e0989153bb99018dc239a4aeaaa2665b816b66d2aa223e3f107d18dac"
Oct 30 22:15:05.813708 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:05.813686    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c_odh-ai-gateway-infra(a0d8db54-4028-4595-8bf7-e44edacf5c20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20"
Oct 30 22:15:05.815151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:05.815131    5031 scope.go:122] "RemoveContainer" containerID="32888afb041d8dce6c1a65bdab8020f3c05378ea87803871f7c50f935649e090"
Oct 30 22:15:05.815493 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:05.815466    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27_odh-ai-gateway-infra(40fe10e0-ecfd-458e-bf13-6343539b47f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7"
Oct 30 22:15:05.817345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:05.817328    5031 scope.go:122] "RemoveContainer" containerID="31e5ec4b36f7218b89339161e082553c21c1e3fdbd44701ee43b81bd06dcbf41"
Oct 30 22:15:05.817617 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:05.817595    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8_odh-ai-gateway-infra(6810b1b3-158a-4c13-b34d-bd87de0806d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6"
Oct 30 22:15:08.073744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:08.073691    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:08.315494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:08.315459    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:09.674867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:09.674831    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:10.921081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:10.920827    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.261109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:11.261018    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:13.665763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:13.665699    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:13.894943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:13.894910    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:14.117644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:14.117610    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:15.065360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:15.065327    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"]
Oct 30 22:15:15.115710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:15.115674    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2a9195-tmltq"]
Oct 30 22:15:15.638660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:15.638623    5031 scope.go:122] "RemoveContainer" containerID="42edd9a5ddafeed2f3e3330316bf9e1aed78d830029c23219a42abcfee7d01ec"
Oct 30 22:15:15.638859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:15.638739    5031 scope.go:122] "RemoveContainer" containerID="31e5ec4b36f7218b89339161e082553c21c1e3fdbd44701ee43b81bd06dcbf41"
Oct 30 22:15:16.105632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.105593    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"]
Oct 30 22:15:16.154041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.154000    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946a92d0-mdxlq"]
Oct 30 22:15:16.202436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.202347    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:16.466508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.466418    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:16.638702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.638659    5031 scope.go:122] "RemoveContainer" containerID="bdccbbc8cadbbef018758ad66e4461869f3b2135b2ff4068c4abb3b567e01b26"
Oct 30 22:15:16.639187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.639163    5031 scope.go:122] "RemoveContainer" containerID="32888afb041d8dce6c1a65bdab8020f3c05378ea87803871f7c50f935649e090"
Oct 30 22:15:16.653142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.653113    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8c4687-ba3b-4509-98e6-907006e836c1" path="/var/lib/kubelet/pods/4c8c4687-ba3b-4509-98e6-907006e836c1/volumes"
Oct 30 22:15:16.653763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:16.653744    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5b4e48-c6ff-4025-9193-9f09f4488bc9" path="/var/lib/kubelet/pods/7d5b4e48-c6ff-4025-9193-9f09f4488bc9/volumes"
Oct 30 22:15:17.002050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.002012    5031 generic.go:356] "Generic (PLEG): container finished" podID="6879a500-443c-4f63-8b36-89e62aa542ed" containerID="67d0b68969315e4382ca27bc3fa1c3a66f742a335aa5fe0d230ad50e9ba615df" exitCode=60
Oct 30 22:15:17.002862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.002322    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" event={"ID":"6879a500-443c-4f63-8b36-89e62aa542ed","Type":"ContainerDied","Data":"67d0b68969315e4382ca27bc3fa1c3a66f742a335aa5fe0d230ad50e9ba615df"}
Oct 30 22:15:17.002862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.002409    5031 scope.go:122] "RemoveContainer" containerID="42edd9a5ddafeed2f3e3330316bf9e1aed78d830029c23219a42abcfee7d01ec"
Oct 30 22:15:17.003188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.003169    5031 scope.go:122] "RemoveContainer" containerID="67d0b68969315e4382ca27bc3fa1c3a66f742a335aa5fe0d230ad50e9ba615df"
Oct 30 22:15:17.003776 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:17.003712    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz_odh-ai-gateway-infra(6879a500-443c-4f63-8b36-89e62aa542ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" podUID="6879a500-443c-4f63-8b36-89e62aa542ed"
Oct 30 22:15:17.007103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.007080    5031 generic.go:356] "Generic (PLEG): container finished" podID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerID="8f9b4fd5f3972b0ba19a5c45226920cf73e198d0833a4f153900cd58cc6fd2e7" exitCode=60
Oct 30 22:15:17.007278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.007148    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" event={"ID":"6810b1b3-158a-4c13-b34d-bd87de0806d6","Type":"ContainerDied","Data":"8f9b4fd5f3972b0ba19a5c45226920cf73e198d0833a4f153900cd58cc6fd2e7"}
Oct 30 22:15:17.007769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.007737    5031 scope.go:122] "RemoveContainer" containerID="8f9b4fd5f3972b0ba19a5c45226920cf73e198d0833a4f153900cd58cc6fd2e7"
Oct 30 22:15:17.008300 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:17.008202    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8_odh-ai-gateway-infra(6810b1b3-158a-4c13-b34d-bd87de0806d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6"
Oct 30 22:15:17.047138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.047081    5031 scope.go:122] "RemoveContainer" containerID="31e5ec4b36f7218b89339161e082553c21c1e3fdbd44701ee43b81bd06dcbf41"
Oct 30 22:15:17.639093 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:17.639064    5031 scope.go:122] "RemoveContainer" containerID="8f7ece2e0989153bb99018dc239a4aeaaa2665b816b66d2aa223e3f107d18dac"
Oct 30 22:15:18.019952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.019915    5031 generic.go:356] "Generic (PLEG): container finished" podID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerID="da5a327fce96e9e44357eb2b46e46bbea3c12cee6188709aed3aaf1299115369" exitCode=60
Oct 30 22:15:18.020107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.019982    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" event={"ID":"40fe10e0-ecfd-458e-bf13-6343539b47f7","Type":"ContainerDied","Data":"da5a327fce96e9e44357eb2b46e46bbea3c12cee6188709aed3aaf1299115369"}
Oct 30 22:15:18.020107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.020040    5031 scope.go:122] "RemoveContainer" containerID="32888afb041d8dce6c1a65bdab8020f3c05378ea87803871f7c50f935649e090"
Oct 30 22:15:18.020554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.020533    5031 scope.go:122] "RemoveContainer" containerID="da5a327fce96e9e44357eb2b46e46bbea3c12cee6188709aed3aaf1299115369"
Oct 30 22:15:18.021022 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:18.020992    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27_odh-ai-gateway-infra(40fe10e0-ecfd-458e-bf13-6343539b47f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7"
Oct 30 22:15:18.025407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.025380    5031 generic.go:356] "Generic (PLEG): container finished" podID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerID="19825f086ea1384154ebecd7d781311c93cd6c970dd79f2229c5bd268e48fde5" exitCode=60
Oct 30 22:15:18.025534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.025456    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" event={"ID":"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e","Type":"ContainerDied","Data":"19825f086ea1384154ebecd7d781311c93cd6c970dd79f2229c5bd268e48fde5"}
Oct 30 22:15:18.026007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.025981    5031 scope.go:122] "RemoveContainer" containerID="19825f086ea1384154ebecd7d781311c93cd6c970dd79f2229c5bd268e48fde5"
Oct 30 22:15:18.026650 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:18.026602    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf_odh-ai-gateway-infra(4092e05f-8cc3-45c6-87a4-4d092b5e0b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e"
Oct 30 22:15:18.044921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.044896    5031 scope.go:122] "RemoveContainer" containerID="bdccbbc8cadbbef018758ad66e4461869f3b2135b2ff4068c4abb3b567e01b26"
Oct 30 22:15:18.095445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.095416    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"]
Oct 30 22:15:18.131866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.131837    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5197cc9-8hn4n"]
Oct 30 22:15:18.171356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.171274    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"]
Oct 30 22:15:18.281947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.277001    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"]
Oct 30 22:15:18.448307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.448176    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:18.505383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.505355    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:18.575718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.575673    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6879a500-443c-4f63-8b36-89e62aa542ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6879a500-443c-4f63-8b36-89e62aa542ed-maas-api-service-ca\") pod \"6879a500-443c-4f63-8b36-89e62aa542ed\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") "
Oct 30 22:15:18.575942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.575732    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6810b1b3-158a-4c13-b34d-bd87de0806d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6810b1b3-158a-4c13-b34d-bd87de0806d6-maas-api-service-ca\") pod \"6810b1b3-158a-4c13-b34d-bd87de0806d6\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") "
Oct 30 22:15:18.575942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.575812    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6879a500-443c-4f63-8b36-89e62aa542ed-kube-api-access-js6pl\" (UniqueName: \"kubernetes.io/projected/6879a500-443c-4f63-8b36-89e62aa542ed-kube-api-access-js6pl\") pod \"6879a500-443c-4f63-8b36-89e62aa542ed\" (UID: \"6879a500-443c-4f63-8b36-89e62aa542ed\") "
Oct 30 22:15:18.575942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.575917    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6810b1b3-158a-4c13-b34d-bd87de0806d6-kube-api-access-4xjkx\" (UniqueName: \"kubernetes.io/projected/6810b1b3-158a-4c13-b34d-bd87de0806d6-kube-api-access-4xjkx\") pod \"6810b1b3-158a-4c13-b34d-bd87de0806d6\" (UID: \"6810b1b3-158a-4c13-b34d-bd87de0806d6\") "
Oct 30 22:15:18.576124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.576100    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6810b1b3-158a-4c13-b34d-bd87de0806d6-maas-api-service-ca" pod "6810b1b3-158a-4c13-b34d-bd87de0806d6" (UID: "6810b1b3-158a-4c13-b34d-bd87de0806d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.576185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.576127    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6879a500-443c-4f63-8b36-89e62aa542ed-maas-api-service-ca" pod "6879a500-443c-4f63-8b36-89e62aa542ed" (UID: "6879a500-443c-4f63-8b36-89e62aa542ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.576687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.576665    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6879a500-443c-4f63-8b36-89e62aa542ed-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.576687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.576687    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6810b1b3-158a-4c13-b34d-bd87de0806d6-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.578224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.578183    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6810b1b3-158a-4c13-b34d-bd87de0806d6-kube-api-access-4xjkx" pod "6810b1b3-158a-4c13-b34d-bd87de0806d6" (UID: "6810b1b3-158a-4c13-b34d-bd87de0806d6"). InnerVolumeSpecName "kube-api-access-4xjkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.578338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.578256    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6879a500-443c-4f63-8b36-89e62aa542ed-kube-api-access-js6pl" pod "6879a500-443c-4f63-8b36-89e62aa542ed" (UID: "6879a500-443c-4f63-8b36-89e62aa542ed"). InnerVolumeSpecName "kube-api-access-js6pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.657092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.657052    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3a6021-8bbf-480a-ace7-98426d786963" path="/var/lib/kubelet/pods/ea3a6021-8bbf-480a-ace7-98426d786963/volumes"
Oct 30 22:15:18.678974 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.678730    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js6pl\" (UniqueName: \"kubernetes.io/projected/6879a500-443c-4f63-8b36-89e62aa542ed-kube-api-access-js6pl\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.678974 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:18.678754    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjkx\" (UniqueName: \"kubernetes.io/projected/6810b1b3-158a-4c13-b34d-bd87de0806d6-kube-api-access-4xjkx\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.042810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.042713    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"
Oct 30 22:15:19.042810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.042729    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz" event={"ID":"6879a500-443c-4f63-8b36-89e62aa542ed","Type":"ContainerDied","Data":"7f53a46cfbf82b686897440c712f24271cbd80a2ebabcb052f4038a7d5efcaa1"}
Oct 30 22:15:19.042810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.042790    5031 scope.go:122] "RemoveContainer" containerID="67d0b68969315e4382ca27bc3fa1c3a66f742a335aa5fe0d230ad50e9ba615df"
Oct 30 22:15:19.045538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.045508    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerID="f946490e200618ec7de98aee946801426ba5c974fdeb9f00c27bad56c9916eec" exitCode=60
Oct 30 22:15:19.045656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.045602    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" event={"ID":"a0d8db54-4028-4595-8bf7-e44edacf5c20","Type":"ContainerDied","Data":"f946490e200618ec7de98aee946801426ba5c974fdeb9f00c27bad56c9916eec"}
Oct 30 22:15:19.046116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.046097    5031 scope.go:122] "RemoveContainer" containerID="f946490e200618ec7de98aee946801426ba5c974fdeb9f00c27bad56c9916eec"
Oct 30 22:15:19.046573 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:19.046549    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c_odh-ai-gateway-infra(a0d8db54-4028-4595-8bf7-e44edacf5c20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20"
Oct 30 22:15:19.050094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.050073    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"
Oct 30 22:15:19.050240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.050112    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8" event={"ID":"6810b1b3-158a-4c13-b34d-bd87de0806d6","Type":"ContainerDied","Data":"45ee87298942e4312f4d78ee91d93ec95ff3fa839f06247fdb1ace2ac9a81573"}
Oct 30 22:15:19.056687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.056671    5031 scope.go:122] "RemoveContainer" containerID="8f7ece2e0989153bb99018dc239a4aeaaa2665b816b66d2aa223e3f107d18dac"
Oct 30 22:15:19.072270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.072253    5031 scope.go:122] "RemoveContainer" containerID="8f9b4fd5f3972b0ba19a5c45226920cf73e198d0833a4f153900cd58cc6fd2e7"
Oct 30 22:15:19.168931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.168894    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"]
Oct 30 22:15:19.227970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.227937    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"]
Oct 30 22:15:19.243431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.239727    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d8837a-29889975-fgct8"]
Oct 30 22:15:19.312634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.312538    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"]
Oct 30 22:15:19.384138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.384112    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:19.475631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.475572    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"]
Oct 30 22:15:19.482539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.482506    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6a0a09-29889975-g6hpz"]
Oct 30 22:15:19.493615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.493583    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40fe10e0-ecfd-458e-bf13-6343539b47f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fe10e0-ecfd-458e-bf13-6343539b47f7-maas-api-service-ca\") pod \"40fe10e0-ecfd-458e-bf13-6343539b47f7\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") "
Oct 30 22:15:19.494012 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.493988    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fe10e0-ecfd-458e-bf13-6343539b47f7-kube-api-access-7bzj8\" (UniqueName: \"kubernetes.io/projected/40fe10e0-ecfd-458e-bf13-6343539b47f7-kube-api-access-7bzj8\") pod \"40fe10e0-ecfd-458e-bf13-6343539b47f7\" (UID: \"40fe10e0-ecfd-458e-bf13-6343539b47f7\") "
Oct 30 22:15:19.496723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.496692    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fe10e0-ecfd-458e-bf13-6343539b47f7-maas-api-service-ca" pod "40fe10e0-ecfd-458e-bf13-6343539b47f7" (UID: "40fe10e0-ecfd-458e-bf13-6343539b47f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.498260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.498178    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fe10e0-ecfd-458e-bf13-6343539b47f7-kube-api-access-7bzj8" pod "40fe10e0-ecfd-458e-bf13-6343539b47f7" (UID: "40fe10e0-ecfd-458e-bf13-6343539b47f7"). InnerVolumeSpecName "kube-api-access-7bzj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.506115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.506092    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:19.570662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.570581    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:19.597198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.597162    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-maas-api-service-ca\") pod \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") "
Oct 30 22:15:19.597534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.597511    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-kube-api-access-gxrlz\" (UniqueName: \"kubernetes.io/projected/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-kube-api-access-gxrlz\") pod \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\" (UID: \"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e\") "
Oct 30 22:15:19.598270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.598233    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fe10e0-ecfd-458e-bf13-6343539b47f7-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.598270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.598258    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bzj8\" (UniqueName: \"kubernetes.io/projected/40fe10e0-ecfd-458e-bf13-6343539b47f7-kube-api-access-7bzj8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.598969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.598945    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-maas-api-service-ca" pod "4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" (UID: "4092e05f-8cc3-45c6-87a4-4d092b5e0b6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.601347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.601317    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-kube-api-access-gxrlz" pod "4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" (UID: "4092e05f-8cc3-45c6-87a4-4d092b5e0b6e"). InnerVolumeSpecName "kube-api-access-gxrlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.699681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.699645    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.699681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.699675    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxrlz\" (UniqueName: \"kubernetes.io/projected/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e-kube-api-access-gxrlz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.818122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:19.818075    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:20.066937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.066900    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27" event={"ID":"40fe10e0-ecfd-458e-bf13-6343539b47f7","Type":"ContainerDied","Data":"21450ee31da9cd43accc6e2ddb1236dabea7ac5300b92aa894b91cbf18b97e18"}
Oct 30 22:15:20.066937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.066924    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"
Oct 30 22:15:20.067256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.066983    5031 scope.go:122] "RemoveContainer" containerID="da5a327fce96e9e44357eb2b46e46bbea3c12cee6188709aed3aaf1299115369"
Oct 30 22:15:20.070372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.070353    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"
Oct 30 22:15:20.070504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.070363    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf" event={"ID":"4092e05f-8cc3-45c6-87a4-4d092b5e0b6e","Type":"ContainerDied","Data":"0acc95979054db534e5071a9ba83698fa42382e0bc78c766147f50c041ba9eb5"}
Oct 30 22:15:20.082300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.082259    5031 scope.go:122] "RemoveContainer" containerID="19825f086ea1384154ebecd7d781311c93cd6c970dd79f2229c5bd268e48fde5"
Oct 30 22:15:20.276235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.259680    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"]
Oct 30 22:15:20.276235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.269950    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9cff60-29889975-khf27"]
Oct 30 22:15:20.398236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.393285    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"]
Oct 30 22:15:20.458111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.457602    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"]
Oct 30 22:15:20.470072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.469916    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889975-v9jwf"]
Oct 30 22:15:20.552407 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:20.552358    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:15:20.649752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.649655    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" path="/var/lib/kubelet/pods/4092e05f-8cc3-45c6-87a4-4d092b5e0b6e/volumes"
Oct 30 22:15:20.650328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.650306    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" path="/var/lib/kubelet/pods/40fe10e0-ecfd-458e-bf13-6343539b47f7/volumes"
Oct 30 22:15:20.651024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.650996    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" path="/var/lib/kubelet/pods/6810b1b3-158a-4c13-b34d-bd87de0806d6/volumes"
Oct 30 22:15:20.651924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.651903    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" path="/var/lib/kubelet/pods/6879a500-443c-4f63-8b36-89e62aa542ed/volumes"
Oct 30 22:15:20.667578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.667556    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:20.777128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.777095    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d8db54-4028-4595-8bf7-e44edacf5c20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8db54-4028-4595-8bf7-e44edacf5c20-maas-api-service-ca\") pod \"a0d8db54-4028-4595-8bf7-e44edacf5c20\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") "
Oct 30 22:15:20.777551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.777166    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8db54-4028-4595-8bf7-e44edacf5c20-kube-api-access-r9kn8\" (UniqueName: \"kubernetes.io/projected/a0d8db54-4028-4595-8bf7-e44edacf5c20-kube-api-access-r9kn8\") pod \"a0d8db54-4028-4595-8bf7-e44edacf5c20\" (UID: \"a0d8db54-4028-4595-8bf7-e44edacf5c20\") "
Oct 30 22:15:20.777598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.777566    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d8db54-4028-4595-8bf7-e44edacf5c20-maas-api-service-ca" pod "a0d8db54-4028-4595-8bf7-e44edacf5c20" (UID: "a0d8db54-4028-4595-8bf7-e44edacf5c20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.777924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.777901    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8db54-4028-4595-8bf7-e44edacf5c20-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.779597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.779572    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8db54-4028-4595-8bf7-e44edacf5c20-kube-api-access-r9kn8" pod "a0d8db54-4028-4595-8bf7-e44edacf5c20" (UID: "a0d8db54-4028-4595-8bf7-e44edacf5c20"). InnerVolumeSpecName "kube-api-access-r9kn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.879820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:20.879783    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kn8\" (UniqueName: \"kubernetes.io/projected/a0d8db54-4028-4595-8bf7-e44edacf5c20-kube-api-access-r9kn8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.063713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:21.063613    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"]
Oct 30 22:15:21.069585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:21.069556    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88c3c4d-s475l"]
Oct 30 22:15:21.083865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:21.083829    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c" event={"ID":"a0d8db54-4028-4595-8bf7-e44edacf5c20","Type":"ContainerDied","Data":"f23fda1ca5b77dd9bbc5d906d5cf4f920416c160dec7d4e96b0ac801bcc7306f"}
Oct 30 22:15:21.083865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:21.083867    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"
Oct 30 22:15:21.084096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:21.083899    5031 scope.go:122] "RemoveContainer" containerID="f946490e200618ec7de98aee946801426ba5c974fdeb9f00c27bad56c9916eec"
Oct 30 22:15:21.259170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:21.259134    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"]
Oct 30 22:15:21.267172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:21.267143    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2182a5-29889975-ww78c"]
Oct 30 22:15:22.651951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:22.651919    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56020270-adad-4456-951a-1d8926a44e11" path="/var/lib/kubelet/pods/56020270-adad-4456-951a-1d8926a44e11/volumes"
Oct 30 22:15:22.652508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:22.652457    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" path="/var/lib/kubelet/pods/a0d8db54-4028-4595-8bf7-e44edacf5c20/volumes"
Oct 30 22:15:24.936614 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:24.936570    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:25.075350 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:25.075306    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"]
Oct 30 22:15:25.077520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:25.077498    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3583a9c1-gs6tn"]
Oct 30 22:15:26.061630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:26.061594    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"]
Oct 30 22:15:26.112200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:26.112162    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7daae72a-dj7vr"]
Oct 30 22:15:26.648329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:26.648288    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5155d09e-f1c7-4236-82ad-51142f45a870" path="/var/lib/kubelet/pods/5155d09e-f1c7-4236-82ad-51142f45a870/volumes"
Oct 30 22:15:26.649034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:26.649015    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd123a07-c372-4eba-9616-e6598c3a47cc" path="/var/lib/kubelet/pods/cd123a07-c372-4eba-9616-e6598c3a47cc/volumes"
Oct 30 22:15:27.060113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:27.060025    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"]
Oct 30 22:15:27.110835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:27.110800    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972f24ad-qqld7"]
Oct 30 22:15:28.648874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:28.648836    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e803b626-79ec-4d21-ad84-677b3f989072" path="/var/lib/kubelet/pods/e803b626-79ec-4d21-ad84-677b3f989072/volumes"
Oct 30 22:15:29.936905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:29.936871    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:34.984885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:34.984850    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:35.061803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:35.061766    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"]
Oct 30 22:15:35.112126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:35.112090    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f8481e9-8pxs9"]
Oct 30 22:15:35.117810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:35.117785    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"]
Oct 30 22:15:35.123455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:35.123431    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30f6ea7a-2q642"]
Oct 30 22:15:36.705236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:36.690465    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f46e03-5714-4437-9a87-9aac2d445a80" path="/var/lib/kubelet/pods/81f46e03-5714-4437-9a87-9aac2d445a80/volumes"
Oct 30 22:15:36.705236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:36.691142    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e5101e-c304-4dcd-901c-030ab139c485" path="/var/lib/kubelet/pods/87e5101e-c304-4dcd-901c-030ab139c485/volumes"
Oct 30 22:15:40.182743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:40.182701    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:41.065084 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:41.064995    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"]
Oct 30 22:15:41.115544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:41.115504    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"]
Oct 30 22:15:41.121041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:41.121014    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f25334f-twjlx"]
Oct 30 22:15:41.126347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:41.126314    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3707f7-snhpb"]
Oct 30 22:15:42.071539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:42.071490    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"]
Oct 30 22:15:42.121621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:42.121578    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a960fb-7tshc"]
Oct 30 22:15:42.650788 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:42.650750    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a6f9ac-660e-4883-a565-9c8d172f77ef" path="/var/lib/kubelet/pods/68a6f9ac-660e-4883-a565-9c8d172f77ef/volumes"
Oct 30 22:15:42.651570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:42.651547    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56" path="/var/lib/kubelet/pods/ae6fb2fd-bed1-4122-a6d7-6a1f3f0e1f56/volumes"
Oct 30 22:15:42.652404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:42.652372    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7cb0d7-3921-45ef-8d9c-11409121c349" path="/var/lib/kubelet/pods/fb7cb0d7-3921-45ef-8d9c-11409121c349/volumes"
Oct 30 22:15:45.063439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:45.063398    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"]
Oct 30 22:15:45.114340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:45.114305    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ec46c-hgfnh"]
Oct 30 22:15:45.336080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:45.335970    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:45.716488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:45.716448    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:45.974314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:45.974187    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:46.648375 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:46.648345    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a10adbb-8b2d-4e02-b916-24143b74a75f" path="/var/lib/kubelet/pods/5a10adbb-8b2d-4e02-b916-24143b74a75f/volumes"
Oct 30 22:15:46.911406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:46.911318    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:47.085597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:47.085562    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"]
Oct 30 22:15:47.124705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:47.124662    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b409a1-r2j6z"]
Oct 30 22:15:47.739752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:47.739671    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:47.968666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:47.968629    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:48.267499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:48.267467    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:48.652435 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:48.652333    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270eaa0b-341c-4a2f-aabc-f086dcba2acc" path="/var/lib/kubelet/pods/270eaa0b-341c-4a2f-aabc-f086dcba2acc/volumes"
Oct 30 22:15:48.916410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:48.916320    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.193295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:49.193174    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.635162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:49.635123    5031 scope.go:122] "RemoveContainer" containerID="d62bbf85b30151de2d553ce032d0905235efdae098f0159f70ade566aeb498fa"
Oct 30 22:15:49.648291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:49.648266    5031 scope.go:122] "RemoveContainer" containerID="0f7f48145bffac39f7b4c0d47112c9df80f3a3c4ecf141c0548779bfdf8d9036"
Oct 30 22:15:49.660600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:49.660578    5031 scope.go:122] "RemoveContainer" containerID="4f43625f81b5aca505ac67c7aa01b6f1bbb2e5ff2e43ae6eb2f54b5e2d9aeec1"
Oct 30 22:15:49.671990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:49.671963    5031 scope.go:122] "RemoveContainer" containerID="69e03dcc85bff1e6b08063c3a5173427e564a6797abffe8fae9f512d3b00ef36"
Oct 30 22:15:49.687914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:49.685033    5031 scope.go:122] "RemoveContainer" containerID="512a60019c0f724febd6aa0a8fe9b8d64800a7710c2ff414b1d087dd39bb8c48"
Oct 30 22:15:50.549921 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:15:50.549879    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:15:50.682300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:50.679041    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:53.025631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:53.025576    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:53.241913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:53.241876    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:55.091830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:55.091788    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"]
Oct 30 22:15:55.142498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:55.142457    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6204f20-frm8b"]
Oct 30 22:15:56.192428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:56.192381    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:56.648411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:56.648376    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2bd04ea-45c1-4f00-b8e7-f842c5ae3949" path="/var/lib/kubelet/pods/f2bd04ea-45c1-4f00-b8e7-f842c5ae3949/volumes"
Oct 30 22:15:57.062043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:57.061961    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"]
Oct 30 22:15:57.112502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:57.112470    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"]
Oct 30 22:15:57.118245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:57.118194    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4705d6b-szkjd"]
Oct 30 22:15:57.123121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:57.123100    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d93752-v7tcf"]
Oct 30 22:15:58.069716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:58.069639    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"]
Oct 30 22:15:58.122090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:58.122053    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d246c60-hg8h8"]
Oct 30 22:15:58.209082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:58.209013    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:58.461045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:58.460969    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:58.652381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:58.652342    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f3490c-1f58-4451-9adb-19ed2dbae0b5" path="/var/lib/kubelet/pods/28f3490c-1f58-4451-9adb-19ed2dbae0b5/volumes"
Oct 30 22:15:58.653099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:58.653074    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423f0bc7-4a12-4d29-9a49-c2bc55b82016" path="/var/lib/kubelet/pods/423f0bc7-4a12-4d29-9a49-c2bc55b82016/volumes"
Oct 30 22:15:58.653747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:15:58.653728    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3" path="/var/lib/kubelet/pods/eaa0e41e-b579-4ac6-ae1f-5e1995af6dc3/volumes"
Oct 30 22:16:01.058666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:01.058618    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"]
Oct 30 22:16:01.109178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:01.109146    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015e1563-2882g"]
Oct 30 22:16:01.329245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:01.329091    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:02.648341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:02.648301    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef016b2-cb83-424d-9f97-06f6bdb78068" path="/var/lib/kubelet/pods/eef016b2-cb83-424d-9f97-06f6bdb78068/volumes"
Oct 30 22:16:03.062735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:03.062649    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"]
Oct 30 22:16:03.112800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:03.112762    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41214da0-bssz4"]
Oct 30 22:16:04.648291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:04.648251    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f63ec52-3309-4a43-bd73-bb7da69b1f19" path="/var/lib/kubelet/pods/2f63ec52-3309-4a43-bd73-bb7da69b1f19/volumes"
Oct 30 22:16:06.504448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:06.504413    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:06.749235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:06.748288    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:06.978459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:06.978425    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:08.066166 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:08.066134    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"]
Oct 30 22:16:08.137386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:08.137348    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace4b013-6dn5w"]
Oct 30 22:16:08.649753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:08.649719    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b02994-fb3b-4d41-95e1-06ad8e06a667" path="/var/lib/kubelet/pods/00b02994-fb3b-4d41-95e1-06ad8e06a667/volumes"
Oct 30 22:16:11.002958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.002916    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"]
Oct 30 22:16:11.003769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003745    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:11.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003770    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:11.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003789    5031 cpu_manager.go:414] "containerMap: removing container" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:11.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003795    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:11.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003804    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:11.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003809    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:11.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003818    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:11.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003822    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003832    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003836    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003849    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003854    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003860    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003864    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003870    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003874    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003883    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003887    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003900    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003904    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003914    5031 cpu_manager.go:414] "containerMap: removing container" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003918    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003948    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:11.004060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.003954    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004074    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004083    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004089    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004095    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004100    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004105    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004113    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004122    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004131    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:11.004581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.004137    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:11.020236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.008263    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.071486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.071433    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"]
Oct 30 22:16:11.118689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.118652    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:11.119685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.119037    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:11.119869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.119145    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:16:11.119978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.119227    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:11.153200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.153167    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.153601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.153571    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.153720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.153708    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6jjk\" (UniqueName: \"kubernetes.io/projected/450ec078-6ffe-43a2-baea-ccb31d4c0689-kube-api-access-v6jjk\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.255181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.255090    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6jjk\" (UniqueName: \"kubernetes.io/projected/450ec078-6ffe-43a2-baea-ccb31d4c0689-kube-api-access-v6jjk\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.255399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.255375    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.255645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.255623    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.256517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.256492    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.258140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.258119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.362178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.362137    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6jjk\" (UniqueName: \"kubernetes.io/projected/450ec078-6ffe-43a2-baea-ccb31d4c0689-kube-api-access-v6jjk\") pod \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:11.646241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:11.646189    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:12.075633 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:16:12.075597    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450ec078_6ffe_43a2_baea_ccb31d4c0689.slice/crio-3f5c36c241fa3ba13cee69786eaf42e30aaf93bcf07e6a7a58bc948e9bf9af3c WatchSource:0}: Error finding container 3f5c36c241fa3ba13cee69786eaf42e30aaf93bcf07e6a7a58bc948e9bf9af3c: Status 404 returned error can't find the container with id 3f5c36c241fa3ba13cee69786eaf42e30aaf93bcf07e6a7a58bc948e9bf9af3c
Oct 30 22:16:12.075633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:12.075612    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"]
Oct 30 22:16:12.317453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:12.317422    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:13.023112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:13.023060    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8" event={"ID":"450ec078-6ffe-43a2-baea-ccb31d4c0689","Type":"ContainerStarted","Data":"3b59226aff82511b3be49342853da75853fd26a911f7f17eaa323bb929db46fb"}
Oct 30 22:16:13.023326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:13.023173    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8" event={"ID":"450ec078-6ffe-43a2-baea-ccb31d4c0689","Type":"ContainerStarted","Data":"3f5c36c241fa3ba13cee69786eaf42e30aaf93bcf07e6a7a58bc948e9bf9af3c"}
Oct 30 22:16:13.145457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:13.145394    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8" podStartSLOduration=3.145379414 podStartE2EDuration="3.145379414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:13.143582476 +0000 UTC m=+1584.667725241" watchObservedRunningTime="2026-10-30 22:16:13.145379414 +0000 UTC m=+1584.669522178"
Oct 30 22:16:14.046048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:14.046009    5031 generic.go:356] "Generic (PLEG): container finished" podID="450ec078-6ffe-43a2-baea-ccb31d4c0689" containerID="3b59226aff82511b3be49342853da75853fd26a911f7f17eaa323bb929db46fb" exitCode=0
Oct 30 22:16:14.046379 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:14.046097    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8" event={"ID":"450ec078-6ffe-43a2-baea-ccb31d4c0689","Type":"ContainerDied","Data":"3b59226aff82511b3be49342853da75853fd26a911f7f17eaa323bb929db46fb"}
Oct 30 22:16:16.065668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:16.065628    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"]
Oct 30 22:16:16.070893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:16.070862    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d390cf28-nhctv"]
Oct 30 22:16:16.659526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:16.659491    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c2273c-603f-402f-bbb9-fbe262767807" path="/var/lib/kubelet/pods/a7c2273c-603f-402f-bbb9-fbe262767807/volumes"
Oct 30 22:16:17.156548 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:17.156508    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"]
Oct 30 22:16:17.165505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:17.165452    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"]
Oct 30 22:16:17.178695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:17.178655    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb6f1cdd-xckjn"]
Oct 30 22:16:17.184566 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:17.184538    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fddba599-8bddx"]
Oct 30 22:16:18.025020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.024834    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:18.065576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.065544    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:18.141335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.141303    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:16:18.141556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.141304    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8" event={"ID":"450ec078-6ffe-43a2-baea-ccb31d4c0689","Type":"ContainerDied","Data":"3f5c36c241fa3ba13cee69786eaf42e30aaf93bcf07e6a7a58bc948e9bf9af3c"}
Oct 30 22:16:18.141556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.141433    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f5c36c241fa3ba13cee69786eaf42e30aaf93bcf07e6a7a58bc948e9bf9af3c"
Oct 30 22:16:18.194425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.194194    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-xks-ca\") pod \"450ec078-6ffe-43a2-baea-ccb31d4c0689\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") "
Oct 30 22:16:18.194425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.194383    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450ec078-6ffe-43a2-baea-ccb31d4c0689-kube-api-access-v6jjk\" (UniqueName: \"kubernetes.io/projected/450ec078-6ffe-43a2-baea-ccb31d4c0689-kube-api-access-v6jjk\") pod \"450ec078-6ffe-43a2-baea-ccb31d4c0689\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") "
Oct 30 22:16:18.195006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.194598    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-openshift-ca\") pod \"450ec078-6ffe-43a2-baea-ccb31d4c0689\" (UID: \"450ec078-6ffe-43a2-baea-ccb31d4c0689\") "
Oct 30 22:16:18.195637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.195612    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-openshift-ca" pod "450ec078-6ffe-43a2-baea-ccb31d4c0689" (UID: "450ec078-6ffe-43a2-baea-ccb31d4c0689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:18.223443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.223403    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450ec078-6ffe-43a2-baea-ccb31d4c0689-kube-api-access-v6jjk" pod "450ec078-6ffe-43a2-baea-ccb31d4c0689" (UID: "450ec078-6ffe-43a2-baea-ccb31d4c0689"). InnerVolumeSpecName "kube-api-access-v6jjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:18.227636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.227600    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-xks-ca" pod "450ec078-6ffe-43a2-baea-ccb31d4c0689" (UID: "450ec078-6ffe-43a2-baea-ccb31d4c0689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:18.297107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.297024    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:18.297107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.297055    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450ec078-6ffe-43a2-baea-ccb31d4c0689-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:18.297107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.297066    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6jjk\" (UniqueName: \"kubernetes.io/projected/450ec078-6ffe-43a2-baea-ccb31d4c0689-kube-api-access-v6jjk\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:18.656106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.656061    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c72c5c-5abe-4297-aa72-8679b1c3d042" path="/var/lib/kubelet/pods/69c72c5c-5abe-4297-aa72-8679b1c3d042/volumes"
Oct 30 22:16:18.656579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:18.656560    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52e2b6e-617f-42d1-9370-c3c293fd14f4" path="/var/lib/kubelet/pods/e52e2b6e-617f-42d1-9370-c3c293fd14f4/volumes"
Oct 30 22:16:19.373768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:19.373735    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:20.093236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:20.092377    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:20.437297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:20.432440    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:20.827131 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:20.827031    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:16:21.076240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.076192    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": read tcp 10.132.0.2:43984->10.132.0.173:15021: read: connection reset by peer" start-of-body=
Oct 30 22:16:21.076374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.076283    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": read tcp 10.132.0.2:43984->10.132.0.173:15021: read: connection reset by peer"
Oct 30 22:16:21.184822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.184792    5031 generic.go:356] "Generic (PLEG): container finished" podID="771f3c50-5456-4960-9921-406fc388b22e" containerID="b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf" exitCode=0
Oct 30 22:16:21.185009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.184873    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" event={"ID":"771f3c50-5456-4960-9921-406fc388b22e","Type":"ContainerDied","Data":"b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf"}
Oct 30 22:16:21.185427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.185409    5031 scope.go:122] "RemoveContainer" containerID="b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf"
Oct 30 22:16:21.187034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.187000    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:21.187091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.187082    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:21.187127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.187114    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:21.428701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.428451    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"]
Oct 30 22:16:21.530810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.530776    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"]
Oct 30 22:16:21.881991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.881787    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:21.989439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.989401    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"]
Oct 30 22:16:21.990319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990294    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:21.990319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990320    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:21.990585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990338    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:21.990585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990346    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:21.990585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990383    5031 cpu_manager.go:414] "containerMap: removing container" podUID="450ec078-6ffe-43a2-baea-ccb31d4c0689" containerName="revoke-keys"
Oct 30 22:16:21.990585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990392    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450ec078-6ffe-43a2-baea-ccb31d4c0689" containerName="revoke-keys"
Oct 30 22:16:21.990585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990415    5031 cpu_manager.go:414] "containerMap: removing container" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:21.990585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990422    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:21.991108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990683    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fe10e0-ecfd-458e-bf13-6343539b47f7" containerName="cleanup"
Oct 30 22:16:21.991108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990701    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4092e05f-8cc3-45c6-87a4-4d092b5e0b6e" containerName="cleanup"
Oct 30 22:16:21.991108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990712    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d8db54-4028-4595-8bf7-e44edacf5c20" containerName="cleanup"
Oct 30 22:16:21.991108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990728    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6879a500-443c-4f63-8b36-89e62aa542ed" containerName="cleanup"
Oct 30 22:16:21.991108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990737    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6810b1b3-158a-4c13-b34d-bd87de0806d6" containerName="cleanup"
Oct 30 22:16:21.991108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.990749    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="450ec078-6ffe-43a2-baea-ccb31d4c0689" containerName="revoke-keys"
Oct 30 22:16:21.993739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:21.993716    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.049104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.049059    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:22.049503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.049481    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:16:22.049728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.049712    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:22.061555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.051955    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:22.069098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.067740    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"]
Oct 30 22:16:22.105690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.105645    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.105991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.105966    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5h22\" (UniqueName: \"kubernetes.io/projected/7088ce87-e083-4269-a75c-02254dcec96a-kube-api-access-v5h22\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.106103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.106085    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.118943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.118910    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:22.212293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.209260    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.212293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.209553    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5h22\" (UniqueName: \"kubernetes.io/projected/7088ce87-e083-4269-a75c-02254dcec96a-kube-api-access-v5h22\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.212293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.209663    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.215010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.214515    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.220778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.219666    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" event={"ID":"771f3c50-5456-4960-9921-406fc388b22e","Type":"ContainerStarted","Data":"4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3"}
Oct 30 22:16:22.224189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.224134    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.288903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.288873    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5h22\" (UniqueName: \"kubernetes.io/projected/7088ce87-e083-4269-a75c-02254dcec96a-kube-api-access-v5h22\") pod \"maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.317656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.317619    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:22.436760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.436280    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:16:22.436760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.436416    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:16:22.436760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.436462    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:16:22.851087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:22.851050    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"]
Oct 30 22:16:23.384237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:23.349017    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk" event={"ID":"7088ce87-e083-4269-a75c-02254dcec96a","Type":"ContainerStarted","Data":"1573b4790df6d928e29baae861e57c0fd36893870d58c9d266c960979e6f2393"}
Oct 30 22:16:23.451237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:23.434532    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:16:23.451237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:23.434588    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:16:24.119821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.119729    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:24.360107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.360064    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk" event={"ID":"7088ce87-e083-4269-a75c-02254dcec96a","Type":"ContainerStarted","Data":"d0b4ecfbea22c0f8e6e4df4ae10f8d74ce44e3401ff7ba2bb099fc6514bb1298"}
Oct 30 22:16:24.405157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.405126    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:24.438994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.438956    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:16:24.439363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.439331    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:16:24.487353 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.487286    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk" podStartSLOduration=3.487266916 podStartE2EDuration="3.487266916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:24.484292059 +0000 UTC m=+1596.008434852" watchObservedRunningTime="2026-10-30 22:16:24.487266916 +0000 UTC m=+1596.011409677"
Oct 30 22:16:24.655714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.655611    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:24.896160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:24.896127    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:25.371364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:25.371331    5031 generic.go:356] "Generic (PLEG): container finished" podID="7088ce87-e083-4269-a75c-02254dcec96a" containerID="d0b4ecfbea22c0f8e6e4df4ae10f8d74ce44e3401ff7ba2bb099fc6514bb1298" exitCode=0
Oct 30 22:16:25.371818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:25.371427    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk" event={"ID":"7088ce87-e083-4269-a75c-02254dcec96a","Type":"ContainerDied","Data":"d0b4ecfbea22c0f8e6e4df4ae10f8d74ce44e3401ff7ba2bb099fc6514bb1298"}
Oct 30 22:16:25.434238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:25.433592    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:16:25.434238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:25.433653    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:16:26.435957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.435920    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:16:26.440785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.440735    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:16:26.441736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.441708    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5" podUID="8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" containerName="payload-processing" containerID="cri-o://abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f" gracePeriod=25
Oct 30 22:16:26.549053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.548766    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv" podUID="ba7ddab7-0613-4b52-9ea6-7f2148d64f11" containerName="payload-pre-processing" containerID="cri-o://cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173" gracePeriod=25
Oct 30 22:16:26.746113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.746080    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:26.867527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.864874    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-openshift-ca\") pod \"7088ce87-e083-4269-a75c-02254dcec96a\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") "
Oct 30 22:16:26.867527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.865151    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7088ce87-e083-4269-a75c-02254dcec96a-kube-api-access-v5h22\" (UniqueName: \"kubernetes.io/projected/7088ce87-e083-4269-a75c-02254dcec96a-kube-api-access-v5h22\") pod \"7088ce87-e083-4269-a75c-02254dcec96a\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") "
Oct 30 22:16:26.867527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.865262    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-xks-ca\") pod \"7088ce87-e083-4269-a75c-02254dcec96a\" (UID: \"7088ce87-e083-4269-a75c-02254dcec96a\") "
Oct 30 22:16:26.867835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.867709    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-openshift-ca" pod "7088ce87-e083-4269-a75c-02254dcec96a" (UID: "7088ce87-e083-4269-a75c-02254dcec96a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:26.870935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.870904    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-xks-ca" pod "7088ce87-e083-4269-a75c-02254dcec96a" (UID: "7088ce87-e083-4269-a75c-02254dcec96a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:26.871338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.871299    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7088ce87-e083-4269-a75c-02254dcec96a-kube-api-access-v5h22" pod "7088ce87-e083-4269-a75c-02254dcec96a" (UID: "7088ce87-e083-4269-a75c-02254dcec96a"). InnerVolumeSpecName "kube-api-access-v5h22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:26.889600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.889087    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:16:26.970863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.969694    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-plugins-config-volume\") pod \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") "
Oct 30 22:16:26.970863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.969910    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-kube-api-access-995rs\" (UniqueName: \"kubernetes.io/projected/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-kube-api-access-995rs\") pod \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\" (UID: \"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0\") "
Oct 30 22:16:26.970863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.970404    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-plugins-config-volume" pod "8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" (UID: "8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:26.970863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.970723    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5h22\" (UniqueName: \"kubernetes.io/projected/7088ce87-e083-4269-a75c-02254dcec96a-kube-api-access-v5h22\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:26.970863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.970748    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:26.970863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.970764    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:26.970863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.970778    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7088ce87-e083-4269-a75c-02254dcec96a-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:26.974233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:26.973232    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-kube-api-access-995rs" pod "8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" (UID: "8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0"). InnerVolumeSpecName "kube-api-access-995rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:27.079234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.052146    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:16:27.079234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.077340    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-995rs\" (UniqueName: \"kubernetes.io/projected/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0-kube-api-access-995rs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:27.193836 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.193790    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-kube-api-access-mw7ts\" (UniqueName: \"kubernetes.io/projected/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-kube-api-access-mw7ts\") pod \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") "
Oct 30 22:16:27.194293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.194270    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-plugins-config-volume\") pod \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\" (UID: \"ba7ddab7-0613-4b52-9ea6-7f2148d64f11\") "
Oct 30 22:16:27.236291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.208545    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-plugins-config-volume" pod "ba7ddab7-0613-4b52-9ea6-7f2148d64f11" (UID: "ba7ddab7-0613-4b52-9ea6-7f2148d64f11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:27.422329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.402587    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-kube-api-access-mw7ts" pod "ba7ddab7-0613-4b52-9ea6-7f2148d64f11" (UID: "ba7ddab7-0613-4b52-9ea6-7f2148d64f11"). InnerVolumeSpecName "kube-api-access-mw7ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:27.422329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.419277    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:27.422329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.419340    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7ts\" (UniqueName: \"kubernetes.io/projected/ba7ddab7-0613-4b52-9ea6-7f2148d64f11-kube-api-access-mw7ts\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:27.448234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.443347    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:16:27.448234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.443409    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:16:27.516562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.503902    5031 generic.go:356] "Generic (PLEG): container finished" podID="ba7ddab7-0613-4b52-9ea6-7f2148d64f11" containerID="cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173" exitCode=0
Oct 30 22:16:27.516562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.504085    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"
Oct 30 22:16:27.516562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.504274    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv" event={"ID":"ba7ddab7-0613-4b52-9ea6-7f2148d64f11","Type":"ContainerDied","Data":"cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173"}
Oct 30 22:16:27.516562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.504331    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv" event={"ID":"ba7ddab7-0613-4b52-9ea6-7f2148d64f11","Type":"ContainerDied","Data":"2c443ccaca5fa37ab9a6c19dc3699487f1b3e458c52d24f332c4c98e8616a27d"}
Oct 30 22:16:27.516562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.504362    5031 scope.go:122] "RemoveContainer" containerID="cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173"
Oct 30 22:16:27.553790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.549697    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk" event={"ID":"7088ce87-e083-4269-a75c-02254dcec96a","Type":"ContainerDied","Data":"1573b4790df6d928e29baae861e57c0fd36893870d58c9d266c960979e6f2393"}
Oct 30 22:16:27.553790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.549764    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1573b4790df6d928e29baae861e57c0fd36893870d58c9d266c960979e6f2393"
Oct 30 22:16:27.553790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.549851    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"
Oct 30 22:16:27.604289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.595347    5031 generic.go:356] "Generic (PLEG): container finished" podID="8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" containerID="abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f" exitCode=0
Oct 30 22:16:27.604289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.595677    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5" event={"ID":"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0","Type":"ContainerDied","Data":"abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f"}
Oct 30 22:16:27.604289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.595745    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5" event={"ID":"8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0","Type":"ContainerDied","Data":"03a12d8347f530f7d5e0510be2f01cc04edcb24d1bb2782fd4f120b2c3b841e5"}
Oct 30 22:16:27.604289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.595854    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"
Oct 30 22:16:27.604289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.604081    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:16:27.655500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.655468    5031 scope.go:122] "RemoveContainer" containerID="cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173"
Oct 30 22:16:27.657031 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:27.656928    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173\": container with ID starting with cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173 not found: ID does not exist" containerID="cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173"
Oct 30 22:16:27.657263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.657074    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173"} err="failed to get container status \"cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173\": rpc error: code = NotFound desc = could not find container \"cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173\": container with ID starting with cab9ede9473584d970e6468055ffa9d35db4a6d82c9ba183e1b00d5615d21173 not found: ID does not exist"
Oct 30 22:16:27.657263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.657102    5031 scope.go:122] "RemoveContainer" containerID="abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f"
Oct 30 22:16:27.681864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.681825    5031 scope.go:122] "RemoveContainer" containerID="abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f"
Oct 30 22:16:27.682310 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:27.682269    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f\": container with ID starting with abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f not found: ID does not exist" containerID="abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f"
Oct 30 22:16:27.682449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.682316    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f"} err="failed to get container status \"abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f\": rpc error: code = NotFound desc = could not find container \"abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f\": container with ID starting with abe17c82b24af62badb5276b73c91964afbb8ae6f4f5097f894b06684338398f not found: ID does not exist"
Oct 30 22:16:27.713496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.713431    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"]
Oct 30 22:16:27.724050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.724002    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv"]
Oct 30 22:16:27.863233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.859527    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"]
Oct 30 22:16:27.878507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:27.869968    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5"]
Oct 30 22:16:28.045234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:28.026080    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.302352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:28.302156    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.676237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:28.673352    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" path="/var/lib/kubelet/pods/8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0/volumes"
Oct 30 22:16:28.676237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:28.674305    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7ddab7-0613-4b52-9ea6-7f2148d64f11" path="/var/lib/kubelet/pods/ba7ddab7-0613-4b52-9ea6-7f2148d64f11/volumes"
Oct 30 22:16:29.060396 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:29.060313    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:16:29.061965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:29.061919    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:16:29.064693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:29.064670    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:16:30.395804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:30.395771    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:32.414261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:32.413532    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"]
Oct 30 22:16:32.972800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:32.972769    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:33.246423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:33.246329    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:34.495301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:34.495257    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.058357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:36.058311    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:37.266513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.266474    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"]
Oct 30 22:16:37.266951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.266709    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy" containerID="cri-o://19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905" gracePeriod=30
Oct 30 22:16:37.317474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.317434    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"]
Oct 30 22:16:37.317674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.317650    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy" containerID="cri-o://9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc" gracePeriod=30
Oct 30 22:16:37.322773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.322742    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"]
Oct 30 22:16:37.322996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.322974    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy" containerID="cri-o://313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773" gracePeriod=30
Oct 30 22:16:37.415390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.415337    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6" podUID="a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" containerName="payload-processing" containerID="cri-o://1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f" gracePeriod=25
Oct 30 22:16:37.750664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.750637    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:16:37.783897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.783864    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" containerID="1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f" exitCode=0
Oct 30 22:16:37.784076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.783906    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"
Oct 30 22:16:37.784076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.783941    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6" event={"ID":"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597","Type":"ContainerDied","Data":"1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f"}
Oct 30 22:16:37.784076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.784000    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6" event={"ID":"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597","Type":"ContainerDied","Data":"55d2e8dedbbdba6aeee831dd1ab022211273008486ff0b4281b1db93ec12e279"}
Oct 30 22:16:37.784076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.784024    5031 scope.go:122] "RemoveContainer" containerID="1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f"
Oct 30 22:16:37.796957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.796937    5031 scope.go:122] "RemoveContainer" containerID="1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f"
Oct 30 22:16:37.797290 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:37.797256    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f\": container with ID starting with 1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f not found: ID does not exist" containerID="1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f"
Oct 30 22:16:37.797351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.797302    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f"} err="failed to get container status \"1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f\": rpc error: code = NotFound desc = could not find container \"1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f\": container with ID starting with 1a39f72a2b2c3d0547e9f292b4b8c108c66ceea04c136723a336eb7fd65ce58f not found: ID does not exist"
Oct 30 22:16:37.814480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.814449    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-kube-api-access-h6fnl\" (UniqueName: \"kubernetes.io/projected/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-kube-api-access-h6fnl\") pod \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") "
Oct 30 22:16:37.814679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.814666    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-plugins-config-volume\") pod \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\" (UID: \"a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597\") "
Oct 30 22:16:37.815123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.815097    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-plugins-config-volume" pod "a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" (UID: "a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:37.815334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.815314    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:37.816861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.816834    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-kube-api-access-h6fnl" pod "a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" (UID: "a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597"). InnerVolumeSpecName "kube-api-access-h6fnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:37.917438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:37.917404    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6fnl\" (UniqueName: \"kubernetes.io/projected/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597-kube-api-access-h6fnl\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:38.214745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:38.214709    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"]
Oct 30 22:16:38.219870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:38.219840    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6"]
Oct 30 22:16:38.648927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:38.648894    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" path="/var/lib/kubelet/pods/a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597/volumes"
Oct 30 22:16:41.184621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:41.184580    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:42.735892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.735866    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:16:42.740547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.740523    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:16:42.765776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.765750    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:16:42.848914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.848879    5031 generic.go:356] "Generic (PLEG): container finished" podID="728354e7-a85a-46da-9c55-e30df62bc50c" containerID="9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc" exitCode=0
Oct 30 22:16:42.849168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.849144    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" event={"ID":"728354e7-a85a-46da-9c55-e30df62bc50c","Type":"ContainerDied","Data":"9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc"}
Oct 30 22:16:42.849299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.849220    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92" event={"ID":"728354e7-a85a-46da-9c55-e30df62bc50c","Type":"ContainerDied","Data":"6dae8a190de9feeee835fd7ac9ef332d5fce6e51d465e99b4cc186a2c1a1e163"}
Oct 30 22:16:42.849299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.849256    5031 scope.go:122] "RemoveContainer" containerID="9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc"
Oct 30 22:16:42.849496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.849479    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"
Oct 30 22:16:42.852816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.852789    5031 generic.go:356] "Generic (PLEG): container finished" podID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerID="19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905" exitCode=0
Oct 30 22:16:42.853437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.853420    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"
Oct 30 22:16:42.855458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.855433    5031 generic.go:356] "Generic (PLEG): container finished" podID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerID="313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773" exitCode=0
Oct 30 22:16:42.855562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.855549    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"
Oct 30 22:16:42.858854 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.858822    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" event={"ID":"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8","Type":"ContainerDied","Data":"19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905"}
Oct 30 22:16:42.858968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.858886    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p" event={"ID":"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8","Type":"ContainerDied","Data":"2d56e28ad386ea2c843156369735d7576aade6e8b6f26597f00239e8654bc407"}
Oct 30 22:16:42.858968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.858913    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" event={"ID":"2a70bb55-c72e-4dfd-9659-8de14b1448ff","Type":"ContainerDied","Data":"313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773"}
Oct 30 22:16:42.858968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.858942    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d" event={"ID":"2a70bb55-c72e-4dfd-9659-8de14b1448ff","Type":"ContainerDied","Data":"f0b9d538d93b518be8686920383643632b2e04d723a75724232dc135bac5e1b7"}
Oct 30 22:16:42.866904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.866873    5031 scope.go:122] "RemoveContainer" containerID="9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc"
Oct 30 22:16:42.867239 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:42.867189    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc\": container with ID starting with 9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc not found: ID does not exist" containerID="9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc"
Oct 30 22:16:42.867303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.867254    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc"} err="failed to get container status \"9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc\": rpc error: code = NotFound desc = could not find container \"9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc\": container with ID starting with 9fa0d2eab52de726e74dc42136c727212a0610cf3c0dc06177bd792ab86901cc not found: ID does not exist"
Oct 30 22:16:42.867303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.867278    5031 scope.go:122] "RemoveContainer" containerID="19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905"
Oct 30 22:16:42.879511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.879489    5031 scope.go:122] "RemoveContainer" containerID="19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905"
Oct 30 22:16:42.879822 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:42.879798    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905\": container with ID starting with 19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905 not found: ID does not exist" containerID="19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905"
Oct 30 22:16:42.879906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.879829    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905"} err="failed to get container status \"19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905\": rpc error: code = NotFound desc = could not find container \"19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905\": container with ID starting with 19c49731d54e5371932b462c16ac68b5fa97fd33c3750874407e9449e3a94905 not found: ID does not exist"
Oct 30 22:16:42.879906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.879846    5031 scope.go:122] "RemoveContainer" containerID="313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773"
Oct 30 22:16:42.891950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.891931    5031 scope.go:122] "RemoveContainer" containerID="313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773"
Oct 30 22:16:42.892307 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:42.892286    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773\": container with ID starting with 313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773 not found: ID does not exist" containerID="313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773"
Oct 30 22:16:42.892399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.892315    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773"} err="failed to get container status \"313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773\": rpc error: code = NotFound desc = could not find container \"313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773\": container with ID starting with 313cf50dc8d964cf356db263ebd37a2b5f976ecc5d28e74971404daa3d60b773 not found: ID does not exist"
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924086    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-credential-socket\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924154    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istiod-ca-cert\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924262    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-istio-token\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-istio-token\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924298    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-envoy\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924332    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-credential-socket\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924369    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-token\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-token\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924410    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-socket\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924452    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-certs\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924485    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-envoy\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924573    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-certs\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924657    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728354e7-a85a-46da-9c55-e30df62bc50c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728354e7-a85a-46da-9c55-e30df62bc50c-istiod-ca-cert\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924690    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-envoy\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924755    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-token\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-token\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924792    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istiod-ca-cert\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924838    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-podinfo\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924919    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-data\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924946    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-kube-api-access-4dhcv\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-kube-api-access-4dhcv\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.924985    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-kube-api-access-z5x7j\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-kube-api-access-z5x7j\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925016    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-kube-api-access-kpmxj\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-kube-api-access-kpmxj\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925045    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/728354e7-a85a-46da-9c55-e30df62bc50c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728354e7-a85a-46da-9c55-e30df62bc50c-istio-podinfo\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925086    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-data\") pod \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\" (UID: \"2a70bb55-c72e-4dfd-9659-8de14b1448ff\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925139    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-credential-socket\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925238    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-certs\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925378    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-socket\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925420    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-data\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925637    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-podinfo\") pod \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\" (UID: \"da9e73e0-ea09-42b6-9c5f-c9c4c44071d8\") "
Oct 30 22:16:42.926983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.925668    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-socket\") pod \"728354e7-a85a-46da-9c55-e30df62bc50c\" (UID: \"728354e7-a85a-46da-9c55-e30df62bc50c\") "
Oct 30 22:16:42.929377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.927725    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-token" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:42.929377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.927968    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-credential-socket" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.929377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.928341    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istiod-ca-cert" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:42.929637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.929612    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-socket" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.929988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.929964    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-data" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.930489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.930468    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-data" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.931534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.930782    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-socket" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.931534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.931157    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728354e7-a85a-46da-9c55-e30df62bc50c-istiod-ca-cert" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:42.931707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.931634    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-certs" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.932819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.932793    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istiod-ca-cert" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:42.936774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.936337    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-credential-socket" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.936995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.936937    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-certs" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.937548 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.937359    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-envoy" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.938062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.937824    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-credential-socket" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.938450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.938398    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-envoy" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.938715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.938445    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-certs" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.938715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.938513    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-socket" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.938715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.938582    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-data" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.939811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.939750    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-podinfo" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:42.940495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.940466    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-kube-api-access-4dhcv" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "kube-api-access-4dhcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:42.940495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.940470    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/728354e7-a85a-46da-9c55-e30df62bc50c-istio-podinfo" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:42.942608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.942580    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-token" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:42.942703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.942585    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-envoy" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:42.952841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.952707    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-kube-api-access-z5x7j" pod "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" (UID: "da9e73e0-ea09-42b6-9c5f-c9c4c44071d8"). InnerVolumeSpecName "kube-api-access-z5x7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:42.952841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.952746    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-kube-api-access-kpmxj" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "kube-api-access-kpmxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:42.952841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.952806    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-istio-token" pod "728354e7-a85a-46da-9c55-e30df62bc50c" (UID: "728354e7-a85a-46da-9c55-e30df62bc50c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:42.956308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:42.956273    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-podinfo" pod "2a70bb55-c72e-4dfd-9659-8de14b1448ff" (UID: "2a70bb55-c72e-4dfd-9659-8de14b1448ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040566    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728354e7-a85a-46da-9c55-e30df62bc50c-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040601    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040610    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040620    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040630    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040639    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040648    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhcv\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-kube-api-access-4dhcv\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040659    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5x7j\" (UniqueName: \"kubernetes.io/projected/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-kube-api-access-z5x7j\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040667    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpmxj\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-kube-api-access-kpmxj\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040675    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728354e7-a85a-46da-9c55-e30df62bc50c-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040684    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040692    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040700    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040708    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040716    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040724    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040731    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040739    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040747    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040754    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728354e7-a85a-46da-9c55-e30df62bc50c-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040762    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040770    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040778    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a70bb55-c72e-4dfd-9659-8de14b1448ff-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040786    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040795    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040803    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728354e7-a85a-46da-9c55-e30df62bc50c-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.040843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.040812    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a70bb55-c72e-4dfd-9659-8de14b1448ff-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:43.366159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.366123    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"]
Oct 30 22:16:43.371105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.371072    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d"]
Oct 30 22:16:43.481455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.481367    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"]
Oct 30 22:16:43.488342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.488314    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p"]
Oct 30 22:16:43.602447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.602284    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"]
Oct 30 22:16:43.608329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:43.608300    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92"]
Oct 30 22:16:44.651619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:44.651573    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" path="/var/lib/kubelet/pods/2a70bb55-c72e-4dfd-9659-8de14b1448ff/volumes"
Oct 30 22:16:44.652347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:44.652325    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" path="/var/lib/kubelet/pods/728354e7-a85a-46da-9c55-e30df62bc50c/volumes"
Oct 30 22:16:44.652875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:44.652853    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" path="/var/lib/kubelet/pods/da9e73e0-ea09-42b6-9c5f-c9c4c44071d8/volumes"
Oct 30 22:16:46.576444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:46.576385    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:47.064372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:47.064337    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"]
Oct 30 22:16:47.116243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:47.116192    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7376483-jtczp"]
Oct 30 22:16:48.073741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:48.073696    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:48.389731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:48.389701    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:48.652365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:48.652280    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9dde23f-fad0-437d-8fb2-21ab8d01ab0d" path="/var/lib/kubelet/pods/d9dde23f-fad0-437d-8fb2-21ab8d01ab0d/volumes"
Oct 30 22:16:49.791782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.791737    5031 scope.go:122] "RemoveContainer" containerID="8d485db4b90f4086dca65ccb1e8ba58d52deaeca19ac5c86fcb8106298253d32"
Oct 30 22:16:49.804182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.804157    5031 scope.go:122] "RemoveContainer" containerID="613b78914ea787d0cc02f1901267558146ddf1e94b0faba8dbf78bfabcc4415f"
Oct 30 22:16:49.816031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.815917    5031 scope.go:122] "RemoveContainer" containerID="8573ff26f70514a3e9a937846a0c27042b47e2a2007c4d316f35afd6e33b0c1b"
Oct 30 22:16:49.828147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.828126    5031 scope.go:122] "RemoveContainer" containerID="ea3cdf2d17ea8352ce04e2c86fd45fce91cbb4db7c00a4bc34217f69cd80fbd7"
Oct 30 22:16:49.839290 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.839271    5031 scope.go:122] "RemoveContainer" containerID="e50fc8b9578544916549927cdce26b1ec68e91dd0c779b1434308250faca5b28"
Oct 30 22:16:49.852721 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.852695    5031 scope.go:122] "RemoveContainer" containerID="791741d047ffbd97d6f9ccbbd8e2737c1221ef8f187c481bc3ea8cf25e529565"
Oct 30 22:16:49.868736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.868714    5031 scope.go:122] "RemoveContainer" containerID="a6ddf494b8e58444c31a028e0e0964d9cec2cd056296232c0aae8826d737fc87"
Oct 30 22:16:49.880124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.880102    5031 scope.go:122] "RemoveContainer" containerID="075e36434ebef4994b1bd1fd2825d6f225be4ee028ce38b2be768c64ae6923ff"
Oct 30 22:16:49.890952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.890926    5031 scope.go:122] "RemoveContainer" containerID="f5772fd9b069a5a0df153538217ee2d8647e1315db35a3e53c3a9236eb7a2aca"
Oct 30 22:16:49.903440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.903416    5031 scope.go:122] "RemoveContainer" containerID="a3742fe9b82851a4c98291f89c694899b2b5d7daa0fdd451af8ad1d66aac9214"
Oct 30 22:16:49.915803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.915779    5031 scope.go:122] "RemoveContainer" containerID="9617f8c05c7d4e5a6ff407ad86f62eafd983d2d5abbd449277e75af5c4d46669"
Oct 30 22:16:49.927358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.927336    5031 scope.go:122] "RemoveContainer" containerID="07a29c923156a9664ce400617b77e458436e35d87b632675eac62788e31cc34f"
Oct 30 22:16:49.966011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.965994    5031 scope.go:122] "RemoveContainer" containerID="51138bb1a84d83d4ce1acea120aef47fe11fd5549acd42e39ce01e1b9763fc4c"
Oct 30 22:16:49.977240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:49.977220    5031 scope.go:122] "RemoveContainer" containerID="4f5c4f6682ed600437ea719d938edce76ee831dba7e8ac9defd8cc3395823d53"
Oct 30 22:16:50.544320 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:16:50.544285    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:16:51.642190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:51.642155    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:56.806658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:56.806625    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:57.059618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:57.059463    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:57.336870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:16:57.336739    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:02.125388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:02.125338    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:03.848092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848055    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"]
Oct 30 22:17:03.848904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848887    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" containerName="payload-processing"
Oct 30 22:17:03.848955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848906    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" containerName="payload-processing"
Oct 30 22:17:03.848955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848936    5031 cpu_manager.go:414] "containerMap: removing container" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy"
Oct 30 22:17:03.848955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848943    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848956    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" containerName="payload-processing"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848961    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" containerName="payload-processing"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848973    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7088ce87-e083-4269-a75c-02254dcec96a" containerName="revoke-keys"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848978    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7088ce87-e083-4269-a75c-02254dcec96a" containerName="revoke-keys"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.848999    5031 cpu_manager.go:414] "containerMap: removing container" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849003    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849018    5031 cpu_manager.go:414] "containerMap: removing container" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849025    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy"
Oct 30 22:17:03.849052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849034    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ba7ddab7-0613-4b52-9ea6-7f2148d64f11" containerName="payload-pre-processing"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849039    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7ddab7-0613-4b52-9ea6-7f2148d64f11" containerName="payload-pre-processing"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849186    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="728354e7-a85a-46da-9c55-e30df62bc50c" containerName="istio-proxy"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849197    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c2d1ac-49b7-41b0-b34d-98bb5c0d0597" containerName="payload-processing"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849203    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7088ce87-e083-4269-a75c-02254dcec96a" containerName="revoke-keys"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849234    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9e73e0-ea09-42b6-9c5f-c9c4c44071d8" containerName="istio-proxy"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849243    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a70bb55-c72e-4dfd-9659-8de14b1448ff" containerName="istio-proxy"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849248    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7ddab7-0613-4b52-9ea6-7f2148d64f11" containerName="payload-pre-processing"
Oct 30 22:17:03.849339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.849256    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b88a3fc-bcaf-46d2-aeee-460ec6df3ed0" containerName="payload-processing"
Oct 30 22:17:03.857127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.857104    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.898567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.898538    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"]
Oct 30 22:17:03.903289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.903270    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.965279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.965243    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"]
Oct 30 22:17:03.991108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991067    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.991371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991132    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.991371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991251    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.991526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991377    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.991526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991415    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.991526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991464    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.991526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991495    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.991753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991543    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4qjw\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-kube-api-access-b4qjw\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.991753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991604    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.991753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991680    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c89898-8f43-479c-af5a-91e8b9661221-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.991951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991746    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.991951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991809    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.991951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991834    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28nj\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-kube-api-access-k28nj\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.991951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991868    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.992179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.991963    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c89898-8f43-479c-af5a-91e8b9661221-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.992179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.992016    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:03.992179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.992055    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:03.992179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:03.992140    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.013627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.013598    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"]
Oct 30 22:17:04.094028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.093985    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094040    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094144    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094177    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094245    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094280    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.094482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094332    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4qjw\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-kube-api-access-b4qjw\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094395    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.094482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094474    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c89898-8f43-479c-af5a-91e8b9661221-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.094697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094507    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.094697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094541    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.094697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094567    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28nj\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-kube-api-access-k28nj\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.094697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094601    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094637    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c89898-8f43-479c-af5a-91e8b9661221-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.094697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094664    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.094697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094690    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.095052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094726    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.095052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094822    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.095052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.094976    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.095241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.095105    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.095241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.095127    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.095346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.095273    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.096403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.095845    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.096403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.095893    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.096403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.096241    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c89898-8f43-479c-af5a-91e8b9661221-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.096403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.096330    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.096403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.096358    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.096797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.096659    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.097357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.097329    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.097631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.097596    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.099057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.099002    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c89898-8f43-479c-af5a-91e8b9661221-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.101073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.101048    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.160175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.160144    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.170916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.170887    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4qjw\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-kube-api-access-b4qjw\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.171362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.171339    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.171362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.171354    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28nj\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-kube-api-access-k28nj\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.173582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.173352    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:04.219920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.219888    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:04.629315 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:04.629283    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6b87c8_d26d_469d_a243_38db94e1cfa6.slice/crio-6191bec34c5d3eabc4992784ec2f1e13f0c15f9acd08985ecb47fb26cef4604e WatchSource:0}: Error finding container 6191bec34c5d3eabc4992784ec2f1e13f0c15f9acd08985ecb47fb26cef4604e: Status 404 returned error can't find the container with id 6191bec34c5d3eabc4992784ec2f1e13f0c15f9acd08985ecb47fb26cef4604e
Oct 30 22:17:04.632113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.632070    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.632246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.632165    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.632320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.632258    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.632691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.632662    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"]
Oct 30 22:17:04.703152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.703125    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"]
Oct 30 22:17:04.703324 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:04.703163    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c89898_8f43_479c_af5a_91e8b9661221.slice/crio-0e7733d6bbd0fdd6febe5002e15af58179d380b4ddcb6233434661d792b47d4c WatchSource:0}: Error finding container 0e7733d6bbd0fdd6febe5002e15af58179d380b4ddcb6233434661d792b47d4c: Status 404 returned error can't find the container with id 0e7733d6bbd0fdd6febe5002e15af58179d380b4ddcb6233434661d792b47d4c
Oct 30 22:17:04.706883 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.706847    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.707007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.706933    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:04.707007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:04.706982    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:05.064004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.063904    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"]
Oct 30 22:17:05.070085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.070058    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9d69f7-5h5rg"]
Oct 30 22:17:05.124334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.124287    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" event={"ID":"76c89898-8f43-479c-af5a-91e8b9661221","Type":"ContainerStarted","Data":"a9352cdac3065d6bdcd805d794f9188720266b59f79569428998a8092a33863d"}
Oct 30 22:17:05.124606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.124363    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" event={"ID":"76c89898-8f43-479c-af5a-91e8b9661221","Type":"ContainerStarted","Data":"0e7733d6bbd0fdd6febe5002e15af58179d380b4ddcb6233434661d792b47d4c"}
Oct 30 22:17:05.127270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.127240    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" event={"ID":"bb6b87c8-d26d-469d-a243-38db94e1cfa6","Type":"ContainerStarted","Data":"6af5361305be68ab660e14a12f8c89f2bf411c7df9c35ca96862104b41a6571e"}
Oct 30 22:17:05.127407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.127301    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" event={"ID":"bb6b87c8-d26d-469d-a243-38db94e1cfa6","Type":"ContainerStarted","Data":"6191bec34c5d3eabc4992784ec2f1e13f0c15f9acd08985ecb47fb26cef4604e"}
Oct 30 22:17:05.174789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.174753    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:05.177409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.177378    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:17:05.177600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.177441    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:17:05.227321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.221578    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:05.243237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.229399    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:17:05.243237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.229475    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:17:05.255972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.255456    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podStartSLOduration=2.255435523 podStartE2EDuration="2.255435523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:05.254169299 +0000 UTC m=+1636.778312065" watchObservedRunningTime="2026-10-30 22:17:05.255435523 +0000 UTC m=+1636.779578290"
Oct 30 22:17:05.385144 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:05.385085    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podStartSLOduration=2.385069771 podStartE2EDuration="2.385069771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:05.38359944 +0000 UTC m=+1636.907742205" watchObservedRunningTime="2026-10-30 22:17:05.385069771 +0000 UTC m=+1636.909212607"
Oct 30 22:17:06.175386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.175354    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:17:06.175794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.175405    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:17:06.227243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.226760    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:17:06.227243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.226843    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:17:06.654258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.650488    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:06.665944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.665906    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25b4244-cc29-41ec-b566-19242a82d8bd" path="/var/lib/kubelet/pods/c25b4244-cc29-41ec-b566-19242a82d8bd/volumes"
Oct 30 22:17:06.979537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.957272    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"]
Oct 30 22:17:06.979537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:06.957543    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy" containerID="cri-o://590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f" gracePeriod=30
Oct 30 22:17:07.212899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.212864    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:17:07.213438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.212924    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:17:07.417238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.410923    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:17:07.417238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.410998    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:17:07.417238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.411291    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"]
Oct 30 22:17:07.463235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.461566    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"]
Oct 30 22:17:07.463235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.462482    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" containerID="cri-o://5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7" gracePeriod=30
Oct 30 22:17:07.472967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.472934    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" containerID="cri-o://66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d" gracePeriod=30
Oct 30 22:17:07.486421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.486388    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"]
Oct 30 22:17:07.486724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.486678    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy" containerID="cri-o://88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86" gracePeriod=30
Oct 30 22:17:07.875234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.870068    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"]
Oct 30 22:17:07.875234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.870432    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" containerID="cri-o://e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70" gracePeriod=30
Oct 30 22:17:07.995865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.995812    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"]
Oct 30 22:17:07.996411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:07.996365    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" containerID="cri-o://72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370" gracePeriod=30
Oct 30 22:17:08.060236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.034740    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"]
Oct 30 22:17:08.060236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.035008    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" containerID="cri-o://917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39" gracePeriod=30
Oct 30 22:17:08.075048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.074916    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"]
Oct 30 22:17:08.075640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.075617    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" containerID="cri-o://50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753" gracePeriod=30
Oct 30 22:17:08.175804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.175694    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:17:08.175804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.175746    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:17:08.230237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.227342    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:17:08.230237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.227426    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:17:08.547312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.546233    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:08.839296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.837597    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:08.925340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.925292    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"]
Oct 30 22:17:08.984347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:08.984309    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"]
Oct 30 22:17:09.062238 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:09.056621    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="450ec078-6ffe-43a2-baea-ccb31d4c0689" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ef8140e-xrzf8"
Oct 30 22:17:09.175963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:09.175922    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:17:09.176157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:09.175985    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:17:09.224236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:09.222131    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:17:09.224236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:09.222196    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:17:09.472557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:09.472399    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:09.472557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:09.472468    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:10.175038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.174991    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:17:10.175263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.175066    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:17:10.240009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.226607    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:17:10.240009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.226669    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 22:17:10.387297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.387258    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"]
Oct 30 22:17:10.387862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.387827    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy" containerID="cri-o://6af5361305be68ab660e14a12f8c89f2bf411c7df9c35ca96862104b41a6571e" gracePeriod=30
Oct 30 22:17:10.403442 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.403413    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"]
Oct 30 22:17:10.403898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.403866    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" containerID="cri-o://4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3" gracePeriod=30
Oct 30 22:17:10.525414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.525285    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:10.570236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.560614    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:10.570236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.544617    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"]
Oct 30 22:17:10.570236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.560978    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy" containerID="cri-o://a9352cdac3065d6bdcd805d794f9188720266b59f79569428998a8092a33863d" gracePeriod=30
Oct 30 22:17:10.596476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.572083    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:10.796298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:10.795491    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450ec078-6ffe-43a2-baea-ccb31d4c0689" path="/var/lib/kubelet/pods/450ec078-6ffe-43a2-baea-ccb31d4c0689/volumes"
Oct 30 22:17:11.093106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.093059    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:11.093307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.093123    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:11.106036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.106001    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:11.106256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.106054    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:11.300935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.300900    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.778182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.778146    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"]
Oct 30 22:17:11.784754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.784732    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.844779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.844726    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-41e4de-openshift-default-dockercfg-tkd9p\""
Oct 30 22:17:11.852194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.852158    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"]
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.894845    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64sd\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-kube-api-access-m64sd\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.894915    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-data\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.895032    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591bc892-852c-416c-9df2-e61a8d9a2015-istio-podinfo\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.895098    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-certs\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.895134    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-istio-token\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.895183    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-socket\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.895286    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591bc892-852c-416c-9df2-e61a8d9a2015-istiod-ca-cert\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.895336    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-envoy\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.899282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.895410    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-credential-socket\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.978178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.978135    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:11.996992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.996959    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-credential-socket\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997026    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64sd\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-kube-api-access-m64sd\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997093    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-data\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997283    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591bc892-852c-416c-9df2-e61a8d9a2015-istio-podinfo\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997366    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-certs\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997407    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-istio-token\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997451    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-credential-socket\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997467    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-socket\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997569    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591bc892-852c-416c-9df2-e61a8d9a2015-istiod-ca-cert\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997628    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-envoy\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.997985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997936    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-socket\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.998087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.997995    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-certs\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.998170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.998116    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-data\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:11.998556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:11.998533    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591bc892-852c-416c-9df2-e61a8d9a2015-istiod-ca-cert\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:12.000234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.000196    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-envoy\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:12.000870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.000846    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591bc892-852c-416c-9df2-e61a8d9a2015-istio-podinfo\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:12.066363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.063368    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64sd\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-kube-api-access-m64sd\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:12.066724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.066644    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-istio-token\") pod \"e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") " pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:12.107773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.107614    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:12.261239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.256180    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.507191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.507150    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 22:17:12.612029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.611988    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-envoy\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612049    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-credential-socket\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612101    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-data\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612166    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-podinfo\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612264    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istiod-ca-cert\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612366    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-certs\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612410    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-token\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-token\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612548    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-kube-api-access-9rdhc\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-kube-api-access-9rdhc\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612596    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-socket\") pod \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\" (UID: \"1b6e59d2-78a1-42a9-af56-e4c01eb00496\") "
Oct 30 22:17:12.612713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612612    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-data" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.612955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.612929    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-credential-socket" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.613201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.613173    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-socket" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.613342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.613314    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istiod-ca-cert" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:12.613959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.613931    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.614083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.613963    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.614083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.613977    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.614083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.613991    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.614083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.614070    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-certs" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.617504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.616497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-kube-api-access-9rdhc" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "kube-api-access-9rdhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:12.617504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.616820    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-podinfo" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:12.618576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.618551    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-token" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:12.619488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.619463    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-envoy" pod "1b6e59d2-78a1-42a9-af56-e4c01eb00496" (UID: "1b6e59d2-78a1-42a9-af56-e4c01eb00496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.682975 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.682893    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:12.683318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.683272    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:12.724048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.720365    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdhc\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-kube-api-access-9rdhc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.732510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.732462    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.732768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.732749    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.732896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.732882    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6e59d2-78a1-42a9-af56-e4c01eb00496-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.733188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.733175    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6e59d2-78a1-42a9-af56-e4c01eb00496-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.768945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.768906    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:17:12.834879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.834848    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-certs\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.835460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.835435    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3357487-cf4c-469f-b600-d02243d486e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3357487-cf4c-469f-b600-d02243d486e2-istio-podinfo\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.835605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.835591    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3357487-cf4c-469f-b600-d02243d486e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3357487-cf4c-469f-b600-d02243d486e2-istiod-ca-cert\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.835823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.835808    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-socket\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.835938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.835925    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-credential-socket\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.836057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.836043    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-data\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.836289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.836269    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-envoy\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.836737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.836713    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-kube-api-access-nfwzd\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-kube-api-access-nfwzd\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.837034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.836982    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-istio-token\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-istio-token\") pod \"f3357487-cf4c-469f-b600-d02243d486e2\" (UID: \"f3357487-cf4c-469f-b600-d02243d486e2\") "
Oct 30 22:17:12.837475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.837449    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-socket" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.838250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.838184    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.838479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.838451    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"]
Oct 30 22:17:12.838786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.838760    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-data" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.838982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.838960    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-credential-socket" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.839306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.839288    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-certs" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.839824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.839798    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3357487-cf4c-469f-b600-d02243d486e2-istiod-ca-cert" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:12.844364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.844341    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-kube-api-access-nfwzd" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "kube-api-access-nfwzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:12.866884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.866349    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-envoy" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:12.874409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.874367    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-istio-token" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:12.877191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.877161    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3357487-cf4c-469f-b600-d02243d486e2-istio-podinfo" pod "f3357487-cf4c-469f-b600-d02243d486e2" (UID: "f3357487-cf4c-469f-b600-d02243d486e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:12.880403 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:12.880373    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591bc892_852c_416c_9df2_e61a8d9a2015.slice/crio-eb4a2465115eab1ceb724c045c500772182120b17c8af19a2f272fd1be1312aa WatchSource:0}: Error finding container eb4a2465115eab1ceb724c045c500772182120b17c8af19a2f272fd1be1312aa: Status 404 returned error can't find the container with id eb4a2465115eab1ceb724c045c500772182120b17c8af19a2f272fd1be1312aa
Oct 30 22:17:12.883656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.883618    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:12.883823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.883712    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:12.883823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.883796    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941869    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941905    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941922    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfwzd\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-kube-api-access-nfwzd\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941937    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3357487-cf4c-469f-b600-d02243d486e2-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941951    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941964    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3357487-cf4c-469f-b600-d02243d486e2-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941979    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3357487-cf4c-469f-b600-d02243d486e2-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.942271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.941992    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3357487-cf4c-469f-b600-d02243d486e2-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:12.957795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.957766    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:17:12.975661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:12.975583    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:17:13.002502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.001020    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" event={"ID":"591bc892-852c-416c-9df2-e61a8d9a2015","Type":"ContainerStarted","Data":"5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490"}
Oct 30 22:17:13.002502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.001076    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" event={"ID":"591bc892-852c-416c-9df2-e61a8d9a2015","Type":"ContainerStarted","Data":"eb4a2465115eab1ceb724c045c500772182120b17c8af19a2f272fd1be1312aa"}
Oct 30 22:17:13.004981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.004950    5031 generic.go:356] "Generic (PLEG): container finished" podID="f3357487-cf4c-469f-b600-d02243d486e2" containerID="66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d" exitCode=0
Oct 30 22:17:13.005073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.005031    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" event={"ID":"f3357487-cf4c-469f-b600-d02243d486e2","Type":"ContainerDied","Data":"66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d"}
Oct 30 22:17:13.005122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.005095    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" event={"ID":"f3357487-cf4c-469f-b600-d02243d486e2","Type":"ContainerDied","Data":"2f2b1ac3347b53d05839d0121c6a43eb2301450a923c1d8c89274d4d3767df82"}
Oct 30 22:17:13.005122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.005119    5031 scope.go:122] "RemoveContainer" containerID="66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d"
Oct 30 22:17:13.005739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.005721    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"
Oct 30 22:17:13.015449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.013924    5031 generic.go:356] "Generic (PLEG): container finished" podID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerID="88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86" exitCode=0
Oct 30 22:17:13.015449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.014005    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" event={"ID":"1f5c548c-8eca-485e-a5c8-22affe05b1f7","Type":"ContainerDied","Data":"88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86"}
Oct 30 22:17:13.015449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.014085    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm" event={"ID":"1f5c548c-8eca-485e-a5c8-22affe05b1f7","Type":"ContainerDied","Data":"5d468ff0a48dbaa760561553ab438840f114e6164ec7d516b1d541d35648bfc8"}
Oct 30 22:17:13.015449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.014162    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"
Oct 30 22:17:13.034650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.018031    5031 generic.go:356] "Generic (PLEG): container finished" podID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerID="590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f" exitCode=0
Oct 30 22:17:13.034650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.018143    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" event={"ID":"1b6e59d2-78a1-42a9-af56-e4c01eb00496","Type":"ContainerDied","Data":"590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f"}
Oct 30 22:17:13.034650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.018245    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl" event={"ID":"1b6e59d2-78a1-42a9-af56-e4c01eb00496","Type":"ContainerDied","Data":"124221ecbcd14f547a4074e961c109d4ec03902cba9dc527230f79b45d1add5a"}
Oct 30 22:17:13.034650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.018346    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"
Oct 30 22:17:13.046818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.046777    5031 generic.go:356] "Generic (PLEG): container finished" podID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerID="5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7" exitCode=0
Oct 30 22:17:13.047048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.046943    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" event={"ID":"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73","Type":"ContainerDied","Data":"5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7"}
Oct 30 22:17:13.047048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047001    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6" event={"ID":"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73","Type":"ContainerDied","Data":"d658ce79851566723eb1da5641f4c06509d55abf1944fe4073fced4471a577e6"}
Oct 30 22:17:13.047237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047085    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"
Oct 30 22:17:13.047686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047663    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-data\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.047820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047735    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-kube-api-access-qs97h\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-kube-api-access-qs97h\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.047820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047753    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-kube-api-access-cchls\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-kube-api-access-cchls\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.047820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047793    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-podinfo\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.047988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047851    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-podinfo\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.047988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047878    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-socket\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.047988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047910    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-envoy\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.047988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047932    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-data\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.047988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.047985    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-envoy\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.048278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048018    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-socket\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.048278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048056    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-certs\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.048278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048081    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-credential-socket\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.048278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048105    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-token\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-token\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.048278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048146    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istiod-ca-cert\") pod \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\" (UID: \"1f5c548c-8eca-485e-a5c8-22affe05b1f7\") "
Oct 30 22:17:13.048278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048264    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-certs\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.048581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048360    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istiod-ca-cert\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.048581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048388    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-credential-socket\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.048581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.048432    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-token\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-token\") pod \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\" (UID: \"e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73\") "
Oct 30 22:17:13.049930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.049897    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-socket" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.054637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.054608    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-certs" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.055055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.054780    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-socket" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.055481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.055456    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-certs" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.055740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.055721    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istiod-ca-cert" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:13.056058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.056038    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-credential-socket" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.056728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.056702    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istiod-ca-cert" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:13.056944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.056924    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-credential-socket" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.062122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.062079    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-kube-api-access-cchls" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "kube-api-access-cchls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.062313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.062287    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-data" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.063896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.063684    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-envoy" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.064235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064031    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-data" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064824    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064853    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064872    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064886    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064900    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064914    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064927    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064940    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064954    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cchls\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-kube-api-access-cchls\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064969    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064983    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.065257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.064996    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.071687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.071449    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-kube-api-access-qs97h" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "kube-api-access-qs97h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.071879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.071857    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-token" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.080589 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.080557    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-envoy" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.082946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.082904    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-token" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.096376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.096122    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-podinfo" pod "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" (UID: "e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:13.098644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.098596    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-podinfo" pod "1f5c548c-8eca-485e-a5c8-22affe05b1f7" (UID: "1f5c548c-8eca-485e-a5c8-22affe05b1f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:13.131231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.125631    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:13.131231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.128408    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:17:13.131231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.128463    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:17:13.167485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.167363    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.167485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.167401    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.167485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.167419    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.167485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.167434    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs97h\" (UniqueName: \"kubernetes.io/projected/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-kube-api-access-qs97h\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.167485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.167448    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5c548c-8eca-485e-a5c8-22affe05b1f7-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.167485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.167461    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.348615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.335147    5031 scope.go:122] "RemoveContainer" containerID="66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d"
Oct 30 22:17:13.348615 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:13.335623    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d\": container with ID starting with 66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d not found: ID does not exist" containerID="66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d"
Oct 30 22:17:13.348615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.335676    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d"} err="failed to get container status \"66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d\": rpc error: code = NotFound desc = could not find container \"66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d\": container with ID starting with 66412e7691bf8ad85597f1666443c57d9fb3cb3ea96c9e85cca487e365238b0d not found: ID does not exist"
Oct 30 22:17:13.348615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.335701    5031 scope.go:122] "RemoveContainer" containerID="88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86"
Oct 30 22:17:13.383832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.378447    5031 scope.go:122] "RemoveContainer" containerID="88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86"
Oct 30 22:17:13.383832 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:13.378893    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86\": container with ID starting with 88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86 not found: ID does not exist" containerID="88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86"
Oct 30 22:17:13.383832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.378926    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86"} err="failed to get container status \"88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86\": rpc error: code = NotFound desc = could not find container \"88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86\": container with ID starting with 88417f30519565a9d38f5ebaeda11a4e637cce95103dab8758a8b78e6148ac86 not found: ID does not exist"
Oct 30 22:17:13.383832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.378963    5031 scope.go:122] "RemoveContainer" containerID="590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f"
Oct 30 22:17:13.418726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.416473    5031 scope.go:122] "RemoveContainer" containerID="4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17"
Oct 30 22:17:13.443628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.443494    5031 scope.go:122] "RemoveContainer" containerID="590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f"
Oct 30 22:17:13.444616 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:13.444180    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f\": container with ID starting with 590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f not found: ID does not exist" containerID="590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f"
Oct 30 22:17:13.444808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.444780    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f"} err="failed to get container status \"590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f\": rpc error: code = NotFound desc = could not find container \"590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f\": container with ID starting with 590a71a99a0fde3b9bb8e7cf5ce0b64656f350bb15adc730f7f0bc318e65948f not found: ID does not exist"
Oct 30 22:17:13.444937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.444924    5031 scope.go:122] "RemoveContainer" containerID="4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17"
Oct 30 22:17:13.445588 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:13.445446    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17\": container with ID starting with 4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17 not found: ID does not exist" containerID="4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17"
Oct 30 22:17:13.445588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.445477    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17"} err="failed to get container status \"4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17\": rpc error: code = NotFound desc = could not find container \"4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17\": container with ID starting with 4260b49b3dd2143247a638f5b22840233dde457ee8eb53d27d46fe38a97f0d17 not found: ID does not exist"
Oct 30 22:17:13.445588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.445498    5031 scope.go:122] "RemoveContainer" containerID="5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7"
Oct 30 22:17:13.487716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.487229    5031 scope.go:122] "RemoveContainer" containerID="5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7"
Oct 30 22:17:13.487716 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:13.487633    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7\": container with ID starting with 5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7 not found: ID does not exist" containerID="5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7"
Oct 30 22:17:13.487716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.487675    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7"} err="failed to get container status \"5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7\": rpc error: code = NotFound desc = could not find container \"5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7\": container with ID starting with 5d68bfce70387a2e3e0ddffa9175226ec64a3c16d287932566d3e214676fc7a7 not found: ID does not exist"
Oct 30 22:17:13.517668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.517386    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.694689    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.694747    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.724877    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-kube-api-access-cpkz9\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-kube-api-access-cpkz9\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.724993    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40284c30-15d8-4040-a338-c77efbdd8e81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40284c30-15d8-4040-a338-c77efbdd8e81-istiod-ca-cert\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.725029    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-envoy\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.725081    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-certs\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.725122    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-socket\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.725249    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-data\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.725293    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-credential-socket\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.725353    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40284c30-15d8-4040-a338-c77efbdd8e81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40284c30-15d8-4040-a338-c77efbdd8e81-istio-podinfo\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.725517    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-istio-token\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-istio-token\") pod \"40284c30-15d8-4040-a338-c77efbdd8e81\" (UID: \"40284c30-15d8-4040-a338-c77efbdd8e81\") "
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.726967    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-certs" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.728806    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.729312    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40284c30-15d8-4040-a338-c77efbdd8e81-istiod-ca-cert" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.729554    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-data" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.729840    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-credential-socket" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.730125    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-socket" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.757390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.751539    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podStartSLOduration=2.751517185 podStartE2EDuration="2.751517185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:13.413409287 +0000 UTC m=+1644.937552052" watchObservedRunningTime="2026-10-30 22:17:13.751517185 +0000 UTC m=+1645.275659950"
Oct 30 22:17:13.766791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.766742    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"]
Oct 30 22:17:13.768130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.768040    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:17:13.780089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.779607    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-envoy" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.780089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.779628    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40284c30-15d8-4040-a338-c77efbdd8e81-istio-podinfo" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:13.780089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.779754    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-kube-api-access-cpkz9" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "kube-api-access-cpkz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.780089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.779793    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-istio-token" pod "40284c30-15d8-4040-a338-c77efbdd8e81" (UID: "40284c30-15d8-4040-a338-c77efbdd8e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.788529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.788498    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm"]
Oct 30 22:17:13.804287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.803766    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:17:13.814950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.814917    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:17:13.829933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.829902    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.829933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.829935    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.829953    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40284c30-15d8-4040-a338-c77efbdd8e81-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.829970    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.829984    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpkz9\" (UniqueName: \"kubernetes.io/projected/40284c30-15d8-4040-a338-c77efbdd8e81-kube-api-access-cpkz9\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.829999    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40284c30-15d8-4040-a338-c77efbdd8e81-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.830014    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.830027    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.830037    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40284c30-15d8-4040-a338-c77efbdd8e81-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.879833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.879769    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"]
Oct 30 22:17:13.884864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.884830    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb"]
Oct 30 22:17:13.930693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930662    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-data\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.930851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930716    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istiod-ca-cert\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.930851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930739    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istiod-ca-cert\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.930851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930760    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-credential-socket\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.930851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930816    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-kube-api-access-98ssh\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-kube-api-access-98ssh\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.930851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930846    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-podinfo\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.931088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930876    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-socket\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.931088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930907    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-kube-api-access-fhqxq\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-kube-api-access-fhqxq\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930926    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-envoy\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.931088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.930987    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-certs\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.931088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931021    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-token\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-token\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.931088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931072    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-credential-socket\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931100    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-credential-socket\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.931348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931122    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-socket\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.931348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931143    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-istio-token\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-istio-token\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931164    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-certs\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.931348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931191    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-token\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-token\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.931348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931258    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-certs\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931286    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-kube-api-access-j6c5x\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-kube-api-access-j6c5x\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.931616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931433    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-data\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.931616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931494    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-podinfo\") pod \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\" (UID: \"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6\") "
Oct 30 22:17:13.931616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931544    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-socket" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.931616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931584    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-envoy\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931665    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0090e71b-05c6-47b4-a868-6f997213b01d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0090e71b-05c6-47b4-a868-6f997213b01d-istio-podinfo\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931711    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-envoy\") pod \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\" (UID: \"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10\") "
Oct 30 22:17:13.931766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931740    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-certs" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.931885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931787    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0090e71b-05c6-47b4-a868-6f997213b01d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0090e71b-05c6-47b4-a868-6f997213b01d-istiod-ca-cert\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931861    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-socket\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.931972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931902    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-data\") pod \"0090e71b-05c6-47b4-a868-6f997213b01d\" (UID: \"0090e71b-05c6-47b4-a868-6f997213b01d\") "
Oct 30 22:17:13.932995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.932970    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.932995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.932998    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.933269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931896    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-credential-socket" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.933269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.931909    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-credential-socket" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.933269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.932047    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-credential-socket" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.933269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.932165    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-data" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.933269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.932407    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istiod-ca-cert" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:13.933269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.932597    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istiod-ca-cert" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:13.934687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.933625    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-socket" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.934687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.933997    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-certs" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.934687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.934184    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-certs" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.936487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.936456    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-podinfo" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:13.938059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.937613    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-data" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.938059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.937725    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-data" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.938059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.937897    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-socket" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.938059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.938028    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-kube-api-access-fhqxq" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "kube-api-access-fhqxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.938388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.938116    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0090e71b-05c6-47b4-a868-6f997213b01d-istiod-ca-cert" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:13.938388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.938193    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-envoy" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.939445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.939391    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-istio-token" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.940280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.940256    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-token" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.940703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.940670    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0090e71b-05c6-47b4-a868-6f997213b01d-istio-podinfo" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:13.940962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.940940    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-kube-api-access-98ssh" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "kube-api-access-98ssh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.941583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.941550    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-kube-api-access-j6c5x" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "kube-api-access-j6c5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.941871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.941828    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-podinfo" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:13.941946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.941880    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-envoy" pod "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" (UID: "7e3fd37c-ce13-4aa4-8af2-b3d89621fc10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.941946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.941891    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-token" pod "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" (UID: "e226a06c-caa4-401c-a4c2-f5d0abb2fcb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.943391 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.943364    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-envoy" pod "0090e71b-05c6-47b4-a868-6f997213b01d" (UID: "0090e71b-05c6-47b4-a868-6f997213b01d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:13.996651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:13.996615    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"]
Oct 30 22:17:14.002038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.002006    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6"]
Oct 30 22:17:14.036123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036085    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036123    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036142    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036157    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036170    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036183    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036196    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036228    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6c5x\" (UniqueName: \"kubernetes.io/projected/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-kube-api-access-j6c5x\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036242    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036258    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036271    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036284    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0090e71b-05c6-47b4-a868-6f997213b01d-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036298    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036311    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0090e71b-05c6-47b4-a868-6f997213b01d-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036324    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036337    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0090e71b-05c6-47b4-a868-6f997213b01d-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036350    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036367    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036381    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036394    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.036399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036407    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ssh\" (UniqueName: \"kubernetes.io/projected/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-kube-api-access-98ssh\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.037395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036422    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.037395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036435    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.037395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036450    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqxq\" (UniqueName: \"kubernetes.io/projected/0090e71b-05c6-47b4-a868-6f997213b01d-kube-api-access-fhqxq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.037395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.036465    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:14.062918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.062880    5031 generic.go:356] "Generic (PLEG): container finished" podID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerID="917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39" exitCode=0
Oct 30 22:17:14.063123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.062940    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"
Oct 30 22:17:14.063123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.063070    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" event={"ID":"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10","Type":"ContainerDied","Data":"917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39"}
Oct 30 22:17:14.063123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.063108    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29" event={"ID":"7e3fd37c-ce13-4aa4-8af2-b3d89621fc10","Type":"ContainerDied","Data":"0bd39c799a97c35a867651f521123771a08433f39890eebf68b232265b0cb3de"}
Oct 30 22:17:14.063460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.063136    5031 scope.go:122] "RemoveContainer" containerID="917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39"
Oct 30 22:17:14.065387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.064323    5031 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:14.065387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.064377    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:14.065867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.065844    5031 generic.go:356] "Generic (PLEG): container finished" podID="40284c30-15d8-4040-a338-c77efbdd8e81" containerID="72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370" exitCode=0
Oct 30 22:17:14.065972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.065902    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"
Oct 30 22:17:14.065972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.065931    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" event={"ID":"40284c30-15d8-4040-a338-c77efbdd8e81","Type":"ContainerDied","Data":"72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370"}
Oct 30 22:17:14.066053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.065994    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj" event={"ID":"40284c30-15d8-4040-a338-c77efbdd8e81","Type":"ContainerDied","Data":"801e2a5ef32d68329c3be43a2d2e153e4339da06521c52819ccb3da7fb0c7088"}
Oct 30 22:17:14.073384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.073362    5031 generic.go:356] "Generic (PLEG): container finished" podID="0090e71b-05c6-47b4-a868-6f997213b01d" containerID="50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753" exitCode=0
Oct 30 22:17:14.073513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.073483    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" event={"ID":"0090e71b-05c6-47b4-a868-6f997213b01d","Type":"ContainerDied","Data":"50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753"}
Oct 30 22:17:14.073627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.073606    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs" event={"ID":"0090e71b-05c6-47b4-a868-6f997213b01d","Type":"ContainerDied","Data":"00c4bc34fce2c2b4caa00d0d64c0f9cc9ec41a79ef2a5ecc7c9bdc4c575ecd27"}
Oct 30 22:17:14.073925 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.073706    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"
Oct 30 22:17:14.076649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.076606    5031 generic.go:356] "Generic (PLEG): container finished" podID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerID="e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70" exitCode=0
Oct 30 22:17:14.076743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.076675    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"
Oct 30 22:17:14.076743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.076699    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" event={"ID":"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6","Type":"ContainerDied","Data":"e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70"}
Oct 30 22:17:14.076743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.076731    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9" event={"ID":"e226a06c-caa4-401c-a4c2-f5d0abb2fcb6","Type":"ContainerDied","Data":"d09145d5dc5d1eca76208ab6b28d40d9dac15f3d73748a84fb8d6e4e08a7cfed"}
Oct 30 22:17:14.082200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.082179    5031 scope.go:122] "RemoveContainer" containerID="917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39"
Oct 30 22:17:14.082586 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:14.082552    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39\": container with ID starting with 917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39 not found: ID does not exist" containerID="917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39"
Oct 30 22:17:14.082979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.082593    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39"} err="failed to get container status \"917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39\": rpc error: code = NotFound desc = could not find container \"917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39\": container with ID starting with 917ec27885faf4b639526587d62fd1c75fd99f001734483346e72f9d16af1d39 not found: ID does not exist"
Oct 30 22:17:14.082979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.082618    5031 scope.go:122] "RemoveContainer" containerID="72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370"
Oct 30 22:17:14.100587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.100560    5031 scope.go:122] "RemoveContainer" containerID="72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370"
Oct 30 22:17:14.101193 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:14.101006    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370\": container with ID starting with 72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370 not found: ID does not exist" containerID="72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370"
Oct 30 22:17:14.101193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.101045    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370"} err="failed to get container status \"72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370\": rpc error: code = NotFound desc = could not find container \"72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370\": container with ID starting with 72abe8c7208743938dbd744e24d52b3ba8924424f034e01e469a45eb96aa1370 not found: ID does not exist"
Oct 30 22:17:14.101193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.101074    5031 scope.go:122] "RemoveContainer" containerID="50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753"
Oct 30 22:17:14.116296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.116259    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:14.121813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.121738    5031 scope.go:122] "RemoveContainer" containerID="50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753"
Oct 30 22:17:14.124681 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:14.124439    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753\": container with ID starting with 50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753 not found: ID does not exist" containerID="50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753"
Oct 30 22:17:14.124681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.124484    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753"} err="failed to get container status \"50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753\": rpc error: code = NotFound desc = could not find container \"50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753\": container with ID starting with 50f532b9dfbab446bd70bea3528585de29ae2cc39796d9b06be7952024dab753 not found: ID does not exist"
Oct 30 22:17:14.124681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.124513    5031 scope.go:122] "RemoveContainer" containerID="e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70"
Oct 30 22:17:14.124681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.124632    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"]
Oct 30 22:17:14.147244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.146904    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl"]
Oct 30 22:17:14.183962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.183891    5031 scope.go:122] "RemoveContainer" containerID="e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70"
Oct 30 22:17:14.184421 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:14.184395    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70\": container with ID starting with e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70 not found: ID does not exist" containerID="e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70"
Oct 30 22:17:14.184539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.184443    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70"} err="failed to get container status \"e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70\": rpc error: code = NotFound desc = could not find container \"e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70\": container with ID starting with e3a4ba679e7c8664719be855fc8c6ee8ee6de41a9cfd1956d0762cc70631ee70 not found: ID does not exist"
Oct 30 22:17:14.442195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.442145    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"]
Oct 30 22:17:14.453864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.453833    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29"]
Oct 30 22:17:14.637497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.637463    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"]
Oct 30 22:17:14.661521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.661486    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" path="/var/lib/kubelet/pods/1b6e59d2-78a1-42a9-af56-e4c01eb00496/volumes"
Oct 30 22:17:14.662072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.662054    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" path="/var/lib/kubelet/pods/1f5c548c-8eca-485e-a5c8-22affe05b1f7/volumes"
Oct 30 22:17:14.662565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.662550    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" path="/var/lib/kubelet/pods/7e3fd37c-ce13-4aa4-8af2-b3d89621fc10/volumes"
Oct 30 22:17:14.663365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.663350    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" path="/var/lib/kubelet/pods/e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73/volumes"
Oct 30 22:17:14.663828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.663814    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3357487-cf4c-469f-b600-d02243d486e2" path="/var/lib/kubelet/pods/f3357487-cf4c-469f-b600-d02243d486e2/volumes"
Oct 30 22:17:14.664157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.664143    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs"]
Oct 30 22:17:14.881071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.881033    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"]
Oct 30 22:17:14.889828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:14.887950    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9"]
Oct 30 22:17:15.059428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.059396    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"]
Oct 30 22:17:15.063534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.063504    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a981d4c-dqznm"]
Oct 30 22:17:15.097888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.097855    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:15.098798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.098776    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:17:15.113645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.113619    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"]
Oct 30 22:17:15.123974 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.123947    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj"]
Oct 30 22:17:15.609041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.609009    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:15.772098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.772074    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:17:15.793345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793314    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-envoy\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793387    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-socket\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793419    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-certs\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793449    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-istio-token\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-istio-token\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793506    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771f3c50-5456-4960-9921-406fc388b22e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771f3c50-5456-4960-9921-406fc388b22e-istiod-ca-cert\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793541    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-data\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793666    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-credential-socket\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793658    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-socket" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.793875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793741    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-certs" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.793929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793896    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-kube-api-access-5nhtp\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-kube-api-access-5nhtp\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.793984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793941    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/771f3c50-5456-4960-9921-406fc388b22e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771f3c50-5456-4960-9921-406fc388b22e-istio-podinfo\") pod \"771f3c50-5456-4960-9921-406fc388b22e\" (UID: \"771f3c50-5456-4960-9921-406fc388b22e\") "
Oct 30 22:17:15.794034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.793982    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-data" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.794034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.794003    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-credential-socket" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.794034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.794001    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771f3c50-5456-4960-9921-406fc388b22e-istiod-ca-cert" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:15.794755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.794600    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.794755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.794623    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.794755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.794634    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771f3c50-5456-4960-9921-406fc388b22e-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.794755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.794642    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.794755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.794650    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.796304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.796198    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-envoy" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:15.796304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.796270    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-istio-token" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:15.796304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.796273    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/771f3c50-5456-4960-9921-406fc388b22e-istio-podinfo" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:15.796450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.796333    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-kube-api-access-5nhtp" pod "771f3c50-5456-4960-9921-406fc388b22e" (UID: "771f3c50-5456-4960-9921-406fc388b22e"). InnerVolumeSpecName "kube-api-access-5nhtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:15.884410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.878456    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:15.920277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.919969    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nhtp\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-kube-api-access-5nhtp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.920277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.919997    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771f3c50-5456-4960-9921-406fc388b22e-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.920277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.920008    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771f3c50-5456-4960-9921-406fc388b22e-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:15.920277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:15.920017    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771f3c50-5456-4960-9921-406fc388b22e-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:16.123791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.123653    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:16.123791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.123728    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"]
Oct 30 22:17:16.128024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.127995    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-781ccf27-nj4sk"]
Oct 30 22:17:16.131088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.131062    5031 generic.go:356] "Generic (PLEG): container finished" podID="771f3c50-5456-4960-9921-406fc388b22e" containerID="4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3" exitCode=0
Oct 30 22:17:16.131241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.131135    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"
Oct 30 22:17:16.131241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.131153    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" event={"ID":"771f3c50-5456-4960-9921-406fc388b22e","Type":"ContainerDied","Data":"4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3"}
Oct 30 22:17:16.131241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.131235    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk" event={"ID":"771f3c50-5456-4960-9921-406fc388b22e","Type":"ContainerDied","Data":"0385b6c7625fe6aaa10acaebac852736375dce6b278939270d02e3d841aded6d"}
Oct 30 22:17:16.131428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.131267    5031 scope.go:122] "RemoveContainer" containerID="4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3"
Oct 30 22:17:16.164232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.158045    5031 scope.go:122] "RemoveContainer" containerID="b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf"
Oct 30 22:17:16.184486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.184458    5031 scope.go:122] "RemoveContainer" containerID="4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3"
Oct 30 22:17:16.185007 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:16.184879    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3\": container with ID starting with 4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3 not found: ID does not exist" containerID="4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3"
Oct 30 22:17:16.185007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.184918    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3"} err="failed to get container status \"4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3\": rpc error: code = NotFound desc = could not find container \"4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3\": container with ID starting with 4430b0093679232637d5c47eb0caf7730f99c3b9a5e98a03f71614f81eb560e3 not found: ID does not exist"
Oct 30 22:17:16.185007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.184945    5031 scope.go:122] "RemoveContainer" containerID="b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf"
Oct 30 22:17:16.186875 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:16.185557    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf\": container with ID starting with b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf not found: ID does not exist" containerID="b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf"
Oct 30 22:17:16.186875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.185587    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf"} err="failed to get container status \"b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf\": rpc error: code = NotFound desc = could not find container \"b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf\": container with ID starting with b1650f0d90fb469164a2681ff0d17c43670240b9cbc0ac5d6f34eca80298e5cf not found: ID does not exist"
Oct 30 22:17:16.345302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.345261    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"]
Oct 30 22:17:16.383295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.376852    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:16.383295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.381453    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk"]
Oct 30 22:17:16.648466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.648366    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" path="/var/lib/kubelet/pods/0090e71b-05c6-47b4-a868-6f997213b01d/volumes"
Oct 30 22:17:16.649068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.649047    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" path="/var/lib/kubelet/pods/40284c30-15d8-4040-a338-c77efbdd8e81/volumes"
Oct 30 22:17:16.649532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.649512    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:16.649793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.649776    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db64b96-3e4e-449c-a52b-4caa77792cc8" path="/var/lib/kubelet/pods/4db64b96-3e4e-449c-a52b-4caa77792cc8/volumes"
Oct 30 22:17:16.650605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.650585    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7088ce87-e083-4269-a75c-02254dcec96a" path="/var/lib/kubelet/pods/7088ce87-e083-4269-a75c-02254dcec96a/volumes"
Oct 30 22:17:16.651029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.651014    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771f3c50-5456-4960-9921-406fc388b22e" path="/var/lib/kubelet/pods/771f3c50-5456-4960-9921-406fc388b22e/volumes"
Oct 30 22:17:16.651483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.651469    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" path="/var/lib/kubelet/pods/e226a06c-caa4-401c-a4c2-f5d0abb2fcb6/volumes"
Oct 30 22:17:16.917928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:16.917831    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.148018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.147976    5031 generic.go:356] "Generic (PLEG): container finished" podID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerID="6af5361305be68ab660e14a12f8c89f2bf411c7df9c35ca96862104b41a6571e" exitCode=0
Oct 30 22:17:17.148388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.148354    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" event={"ID":"bb6b87c8-d26d-469d-a243-38db94e1cfa6","Type":"ContainerDied","Data":"6af5361305be68ab660e14a12f8c89f2bf411c7df9c35ca96862104b41a6571e"}
Oct 30 22:17:17.151713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.151680    5031 generic.go:356] "Generic (PLEG): container finished" podID="76c89898-8f43-479c-af5a-91e8b9661221" containerID="a9352cdac3065d6bdcd805d794f9188720266b59f79569428998a8092a33863d" exitCode=0
Oct 30 22:17:17.151843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.151798    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" event={"ID":"76c89898-8f43-479c-af5a-91e8b9661221","Type":"ContainerDied","Data":"a9352cdac3065d6bdcd805d794f9188720266b59f79569428998a8092a33863d"}
Oct 30 22:17:17.183458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.183359    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.300451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.300420    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.349654    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-certs\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.349755    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-kube-api-access-k28nj\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-kube-api-access-k28nj\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.349809    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-credential-socket\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.349839    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-data\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.349900    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c89898-8f43-479c-af5a-91e8b9661221-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c89898-8f43-479c-af5a-91e8b9661221-istio-podinfo\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.349930    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-envoy\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.349971    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-istio-token\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-istio-token\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.350019    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-socket\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.354993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.350172    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c89898-8f43-479c-af5a-91e8b9661221-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c89898-8f43-479c-af5a-91e8b9661221-istiod-ca-cert\") pod \"76c89898-8f43-479c-af5a-91e8b9661221\" (UID: \"76c89898-8f43-479c-af5a-91e8b9661221\") "
Oct 30 22:17:17.356136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.356094    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-certs" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.357137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.357115    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c89898-8f43-479c-af5a-91e8b9661221-istiod-ca-cert" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:17.358046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.357857    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-credential-socket" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.358509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.358225    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-data" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.359139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.359118    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-socket" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.361756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.361709    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-kube-api-access-k28nj" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "kube-api-access-k28nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.363417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.363379    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-istio-token" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.364192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.364140    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c89898-8f43-479c-af5a-91e8b9661221-istio-podinfo" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:17.364987 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.363917    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-envoy" pod "76c89898-8f43-479c-af5a-91e8b9661221" (UID: "76c89898-8f43-479c-af5a-91e8b9661221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.395125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.395079    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:17.452685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452579    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-socket\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.452685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452672    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-envoy\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.452937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452743    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-podinfo\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.452937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452791    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-token\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-token\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.452937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452828    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istiod-ca-cert\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.452937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452871    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-credential-socket\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.453141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452952    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-certs\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.453141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.452992    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-data\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.453141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453122    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-kube-api-access-b4qjw\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-kube-api-access-b4qjw\") pod \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\" (UID: \"bb6b87c8-d26d-469d-a243-38db94e1cfa6\") "
Oct 30 22:17:17.453843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453818    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c89898-8f43-479c-af5a-91e8b9661221-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.453843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453845    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453860    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28nj\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-kube-api-access-k28nj\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453874    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453887    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453899    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c89898-8f43-479c-af5a-91e8b9661221-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453912    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453924    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c89898-8f43-479c-af5a-91e8b9661221-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453936    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c89898-8f43-479c-af5a-91e8b9661221-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.454041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.453927    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-credential-socket" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.454424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.454088    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-socket" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.454424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.454382    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istiod-ca-cert" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:17.454815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.454691    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-data" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.455116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.455083    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-certs" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.457398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.457359    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-envoy" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.458910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.458886    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-kube-api-access-b4qjw" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "kube-api-access-b4qjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.459156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.459130    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-podinfo" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:17.463659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.463612    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-token" pod "bb6b87c8-d26d-469d-a243-38db94e1cfa6" (UID: "bb6b87c8-d26d-469d-a243-38db94e1cfa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.556101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556064    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556103    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556119    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4qjw\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-kube-api-access-b4qjw\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556134    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556148    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556163    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556177    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556193    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6b87c8-d26d-469d-a243-38db94e1cfa6-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.556390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.556243    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6b87c8-d26d-469d-a243-38db94e1cfa6-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.622357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.622313    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"]
Oct 30 22:17:17.623410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623385    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy"
Oct 30 22:17:17.623410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623411    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623425    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623433    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623466    5031 cpu_manager.go:414] "containerMap: removing container" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623473    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623496    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623505    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623516    5031 cpu_manager.go:414] "containerMap: removing container" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623522    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623535    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623542    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623552    5031 cpu_manager.go:414] "containerMap: removing container" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623559    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623571    5031 cpu_manager.go:414] "containerMap: removing container" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy"
Oct 30 22:17:17.623580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623578    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623598    5031 cpu_manager.go:414] "containerMap: removing container" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623604    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623626    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623633    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623642    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623649    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623666    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623673    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623840    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623854    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3fd37c-ce13-4aa4-8af2-b3d89621fc10" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623869    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="40284c30-15d8-4040-a338-c77efbdd8e81" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623878    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e337b5e1-86c2-4f6e-bb6c-afcfa9e47f73" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623888    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623900    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623909    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c89898-8f43-479c-af5a-91e8b9661221" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623921    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0090e71b-05c6-47b4-a868-6f997213b01d" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623929    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5c548c-8eca-485e-a5c8-22affe05b1f7" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623940    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3357487-cf4c-469f-b600-d02243d486e2" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623952    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e226a06c-caa4-401c-a4c2-f5d0abb2fcb6" containerName="istio-proxy"
Oct 30 22:17:17.624043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.623965    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="771f3c50-5456-4960-9921-406fc388b22e" containerName="istio-proxy"
Oct 30 22:17:17.628637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.628607    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.685724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.685005    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b55a43-openshift-default-dockercfg-tsbxs\""
Oct 30 22:17:17.686891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.686844    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-credential-socket\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.687032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.686962    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-socket\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.687109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.687031    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895bc537-7f1c-4950-bfa0-b89ae9531f95-istiod-ca-cert\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.687176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.687107    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-data\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.690126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.689564    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-certs\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.690126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.689662    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-token\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.690126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.689686    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-podinfo\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.690126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.689834    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd6vp\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-kube-api-access-kd6vp\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.690126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.689895    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-envoy\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.761324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.761290    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"]
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.791711    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-data\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.792269    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-certs\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.792480    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-token\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.792511    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-podinfo\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.792768    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-data\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.792904    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd6vp\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-kube-api-access-kd6vp\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.792964    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-envoy\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.793023    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-credential-socket\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.793075    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-socket\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.793136    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895bc537-7f1c-4950-bfa0-b89ae9531f95-istiod-ca-cert\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.794195    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-socket\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.794409    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-credential-socket\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.795065    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895bc537-7f1c-4950-bfa0-b89ae9531f95-istiod-ca-cert\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.796045    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-certs\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.798164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.796892    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.802578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.802519    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-envoy\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.803228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.803189    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-podinfo\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.876245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.860661    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd6vp\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-kube-api-access-kd6vp\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:17.876245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:17.861867    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-token\") pod \"e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") " pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:18.002868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.002777    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:18.139917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.139812    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:18.168740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.168703    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"
Oct 30 22:17:18.168740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.168731    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98" event={"ID":"76c89898-8f43-479c-af5a-91e8b9661221","Type":"ContainerDied","Data":"0e7733d6bbd0fdd6febe5002e15af58179d380b4ddcb6233434661d792b47d4c"}
Oct 30 22:17:18.168994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.168801    5031 scope.go:122] "RemoveContainer" containerID="a9352cdac3065d6bdcd805d794f9188720266b59f79569428998a8092a33863d"
Oct 30 22:17:18.170812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.170790    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx" event={"ID":"bb6b87c8-d26d-469d-a243-38db94e1cfa6","Type":"ContainerDied","Data":"6191bec34c5d3eabc4992784ec2f1e13f0c15f9acd08985ecb47fb26cef4604e"}
Oct 30 22:17:18.170944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.170825    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"
Oct 30 22:17:18.186445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.186425    5031 scope.go:122] "RemoveContainer" containerID="6af5361305be68ab660e14a12f8c89f2bf411c7df9c35ca96862104b41a6571e"
Oct 30 22:17:18.358568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.358529    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"]
Oct 30 22:17:18.364524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.364496    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98"]
Oct 30 22:17:18.389178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.389133    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:18.485563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.485530    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"]
Oct 30 22:17:18.497405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.497376    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx"]
Oct 30 22:17:18.533092 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:18.533065    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895bc537_7f1c_4950_bfa0_b89ae9531f95.slice/crio-cf47ac9911ff499c3ff4550a22f98ea4e474b2c00d981cc8f90c5c6cf1bfc766 WatchSource:0}: Error finding container cf47ac9911ff499c3ff4550a22f98ea4e474b2c00d981cc8f90c5c6cf1bfc766: Status 404 returned error can't find the container with id cf47ac9911ff499c3ff4550a22f98ea4e474b2c00d981cc8f90c5c6cf1bfc766
Oct 30 22:17:18.534102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.534049    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"]
Oct 30 22:17:18.536759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.536719    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:18.536852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.536809    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:18.536903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.536859    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:18.652890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.652846    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c89898-8f43-479c-af5a-91e8b9661221" path="/var/lib/kubelet/pods/76c89898-8f43-479c-af5a-91e8b9661221/volumes"
Oct 30 22:17:18.653626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:18.653602    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6b87c8-d26d-469d-a243-38db94e1cfa6" path="/var/lib/kubelet/pods/bb6b87c8-d26d-469d-a243-38db94e1cfa6/volumes"
Oct 30 22:17:19.185662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:19.185611    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" event={"ID":"895bc537-7f1c-4950-bfa0-b89ae9531f95","Type":"ContainerStarted","Data":"c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2"}
Oct 30 22:17:19.186133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:19.185678    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" event={"ID":"895bc537-7f1c-4950-bfa0-b89ae9531f95","Type":"ContainerStarted","Data":"cf47ac9911ff499c3ff4550a22f98ea4e474b2c00d981cc8f90c5c6cf1bfc766"}
Oct 30 22:17:19.249644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:19.249609    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:19.328496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:19.328420    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podStartSLOduration=2.328397216 podStartE2EDuration="2.328397216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:19.309271765 +0000 UTC m=+1650.833414528" watchObservedRunningTime="2026-10-30 22:17:19.328397216 +0000 UTC m=+1650.852539981"
Oct 30 22:17:19.481399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:19.481307    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:19.896471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:19.896430    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.003510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.003473    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:20.010754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.010723    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:20.198097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.197980    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:20.200173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.199929    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:17:20.288822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.288786    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.436049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.436016    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"]
Oct 30 22:17:20.436771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.436753    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy"
Oct 30 22:17:20.436771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.436771    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy"
Oct 30 22:17:20.436964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.436906    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6e59d2-78a1-42a9-af56-e4c01eb00496" containerName="istio-proxy"
Oct 30 22:17:20.442018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.441995    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.500550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.500518    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"]
Oct 30 22:17:20.501008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.500986    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-41e4de\""
Oct 30 22:17:20.535841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.535804    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"]
Oct 30 22:17:20.571255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.563464    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:20.571255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.568075    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4428c1b-22ee-4577-8786-a9c399f57154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.571255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.568118    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/e4428c1b-22ee-4577-8786-a9c399f57154-kube-api-access-6rjcc\") pod \"payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.606255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.597688    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-41e4de-dockercfg-8bcnp\""
Oct 30 22:17:20.606255 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:20.597856    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:17:20.632265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.632155    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"]
Oct 30 22:17:20.684384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.681616    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tprm\" (UniqueName: \"kubernetes.io/projected/98118413-b24e-4f80-8fe9-15a09b89ef98-kube-api-access-2tprm\") pod \"payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:20.684384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.681688    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98118413-b24e-4f80-8fe9-15a09b89ef98-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:20.684384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.681762    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4428c1b-22ee-4577-8786-a9c399f57154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.684384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.681809    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/e4428c1b-22ee-4577-8786-a9c399f57154-kube-api-access-6rjcc\") pod \"payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.684384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.683375    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4428c1b-22ee-4577-8786-a9c399f57154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.747707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.747675    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/e4428c1b-22ee-4577-8786-a9c399f57154-kube-api-access-6rjcc\") pod \"payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.760200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.760167    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:20.784113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.784077    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tprm\" (UniqueName: \"kubernetes.io/projected/98118413-b24e-4f80-8fe9-15a09b89ef98-kube-api-access-2tprm\") pod \"payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:20.784353 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.784142    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98118413-b24e-4f80-8fe9-15a09b89ef98-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:20.785115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.785094    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98118413-b24e-4f80-8fe9-15a09b89ef98-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:20.852831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:20.852799    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tprm\" (UniqueName: \"kubernetes.io/projected/98118413-b24e-4f80-8fe9-15a09b89ef98-kube-api-access-2tprm\") pod \"payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:21.019814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:21.019724    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:21.201758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:21.201729    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"]
Oct 30 22:17:21.203143 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:21.203117    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4428c1b_22ee_4577_8786_a9c399f57154.slice/crio-f04cb7d1b5420fb9efdb4a3f2cc439a9b06e52faa532f7144788ddaa13316779 WatchSource:0}: Error finding container f04cb7d1b5420fb9efdb4a3f2cc439a9b06e52faa532f7144788ddaa13316779: Status 404 returned error can't find the container with id f04cb7d1b5420fb9efdb4a3f2cc439a9b06e52faa532f7144788ddaa13316779
Oct 30 22:17:21.449640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:21.449597    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"]
Oct 30 22:17:21.455427 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:21.455395    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98118413_b24e_4f80_8fe9_15a09b89ef98.slice/crio-5ccd7ec94f0847cd5954f19e824d8948ac651b3463a6d72894113a5d8fd75fb8 WatchSource:0}: Error finding container 5ccd7ec94f0847cd5954f19e824d8948ac651b3463a6d72894113a5d8fd75fb8: Status 404 returned error can't find the container with id 5ccd7ec94f0847cd5954f19e824d8948ac651b3463a6d72894113a5d8fd75fb8
Oct 30 22:17:21.985200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:21.985132    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:22.249860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.248247    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96" event={"ID":"e4428c1b-22ee-4577-8786-a9c399f57154","Type":"ContainerStarted","Data":"8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd"}
Oct 30 22:17:22.249860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.248321    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96" event={"ID":"e4428c1b-22ee-4577-8786-a9c399f57154","Type":"ContainerStarted","Data":"f04cb7d1b5420fb9efdb4a3f2cc439a9b06e52faa532f7144788ddaa13316779"}
Oct 30 22:17:22.249860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.249483    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:22.257292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.253725    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb" event={"ID":"98118413-b24e-4f80-8fe9-15a09b89ef98","Type":"ContainerStarted","Data":"26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0"}
Oct 30 22:17:22.257292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.253768    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:22.257292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.253783    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb" event={"ID":"98118413-b24e-4f80-8fe9-15a09b89ef98","Type":"ContainerStarted","Data":"5ccd7ec94f0847cd5954f19e824d8948ac651b3463a6d72894113a5d8fd75fb8"}
Oct 30 22:17:22.312912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.312481    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:22.330061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.330021    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:17:22.386663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.386583    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96" podStartSLOduration=2.386563603 podStartE2EDuration="2.386563603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:22.380582822 +0000 UTC m=+1653.904725586" watchObservedRunningTime="2026-10-30 22:17:22.386563603 +0000 UTC m=+1653.910706370"
Oct 30 22:17:22.523516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.523400    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb" podStartSLOduration=2.5233801209999998 podStartE2EDuration="2.523380121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:22.516833595 +0000 UTC m=+1654.040976371" watchObservedRunningTime="2026-10-30 22:17:22.523380121 +0000 UTC m=+1654.047522886"
Oct 30 22:17:22.606603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.606568    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:22.841459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:22.841374    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:25.795513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:25.795441    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:26.032394 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.032361    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:26.142009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.141968    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"]
Oct 30 22:17:26.162310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.162275    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.211965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.211925    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"]
Oct 30 22:17:26.256007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.255966    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b55a43-dockercfg-49klb\""
Oct 30 22:17:26.256261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.256018    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b55a43\""
Oct 30 22:17:26.325911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.315740    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc7p6\" (UniqueName: \"kubernetes.io/projected/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-kube-api-access-wc7p6\") pod \"payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.325911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.315870    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.333892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.333857    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"]
Oct 30 22:17:26.340394 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.340367    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.395558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.395384    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"]
Oct 30 22:17:26.419519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.419484    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.419737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.419619    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba87d160-7551-4ea4-a296-ebf9155b5b5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.419737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.419722    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5lg\" (UniqueName: \"kubernetes.io/projected/ba87d160-7551-4ea4-a296-ebf9155b5b5b-kube-api-access-gf5lg\") pod \"payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.419882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.419767    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc7p6\" (UniqueName: \"kubernetes.io/projected/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-kube-api-access-wc7p6\") pod \"payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.420766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.420743    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.486276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.485502    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc7p6\" (UniqueName: \"kubernetes.io/projected/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-kube-api-access-wc7p6\") pod \"payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.493278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.493246    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:26.521591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.521553    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba87d160-7551-4ea4-a296-ebf9155b5b5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.521778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.521712    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5lg\" (UniqueName: \"kubernetes.io/projected/ba87d160-7551-4ea4-a296-ebf9155b5b5b-kube-api-access-gf5lg\") pod \"payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.522454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.522433    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba87d160-7551-4ea4-a296-ebf9155b5b5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.587714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.587670    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5lg\" (UniqueName: \"kubernetes.io/projected/ba87d160-7551-4ea4-a296-ebf9155b5b5b-kube-api-access-gf5lg\") pod \"payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.660055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.658664    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:26.914451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:26.914422    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"]
Oct 30 22:17:26.916427 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:26.916392    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c7f8d7_3827_4eb9_acde_d99f7c7b9e0e.slice/crio-d4ceddef590bc426d992954abca17333c93a363caddc895931c3ce3ca9621a99 WatchSource:0}: Error finding container d4ceddef590bc426d992954abca17333c93a363caddc895931c3ce3ca9621a99: Status 404 returned error can't find the container with id d4ceddef590bc426d992954abca17333c93a363caddc895931c3ce3ca9621a99
Oct 30 22:17:27.093647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.093613    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"]
Oct 30 22:17:27.093837 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:27.093810    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba87d160_7551_4ea4_a296_ebf9155b5b5b.slice/crio-278792cff09ad16329fb2928acaf20f0fbaa8f858dc7f662c5ebe1ada0f68b78 WatchSource:0}: Error finding container 278792cff09ad16329fb2928acaf20f0fbaa8f858dc7f662c5ebe1ada0f68b78: Status 404 returned error can't find the container with id 278792cff09ad16329fb2928acaf20f0fbaa8f858dc7f662c5ebe1ada0f68b78
Oct 30 22:17:27.442859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.442728    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g" event={"ID":"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e","Type":"ContainerStarted","Data":"1c538834c1fd158068dc991e97fe09fcfec45174ca27b5a5586449155a727955"}
Oct 30 22:17:27.442859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.442802    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g" event={"ID":"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e","Type":"ContainerStarted","Data":"d4ceddef590bc426d992954abca17333c93a363caddc895931c3ce3ca9621a99"}
Oct 30 22:17:27.442859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.442836    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:27.445059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.445026    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k" event={"ID":"ba87d160-7551-4ea4-a296-ebf9155b5b5b","Type":"ContainerStarted","Data":"fafce537e2c070101c8fdd205cbc95c0949a89ef44dff543acac85c9eb28ce28"}
Oct 30 22:17:27.445260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.445077    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k" event={"ID":"ba87d160-7551-4ea4-a296-ebf9155b5b5b","Type":"ContainerStarted","Data":"278792cff09ad16329fb2928acaf20f0fbaa8f858dc7f662c5ebe1ada0f68b78"}
Oct 30 22:17:27.445260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.445103    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:27.563671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.563616    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g" podStartSLOduration=1.5636027590000001 podStartE2EDuration="1.563602759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:27.561526842 +0000 UTC m=+1659.085669606" watchObservedRunningTime="2026-10-30 22:17:27.563602759 +0000 UTC m=+1659.087745520"
Oct 30 22:17:27.685329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.685269    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k" podStartSLOduration=1.685249887 podStartE2EDuration="1.685249887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:27.680700187 +0000 UTC m=+1659.204842951" watchObservedRunningTime="2026-10-30 22:17:27.685249887 +0000 UTC m=+1659.209392650"
Oct 30 22:17:27.710717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:27.710621    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.208510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.208469    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"]
Oct 30 22:17:28.214453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.214424    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.300235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.294239    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-512caa-openshift-default-dockercfg-gbsns\""
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.312719    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnm9v\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-kube-api-access-gnm9v\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.312783    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istiod-ca-cert\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.312905    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-envoy\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.313004    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-token\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.313100    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-podinfo\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.313269    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-credential-socket\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.313327    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-data\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.313372    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-certs\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.315240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.313437    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-socket\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.326261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.316928    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"]
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.416534    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-socket\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.417043    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-socket\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.417135    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnm9v\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-kube-api-access-gnm9v\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.417179    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istiod-ca-cert\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.417655    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-envoy\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.417750    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-token\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.417849    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-podinfo\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.417973    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-credential-socket\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.418006    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-data\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.418043    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-certs\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.418309    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istiod-ca-cert\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.418779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.418399    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-certs\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.419606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.418838    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-credential-socket\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.419606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.419325    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-data\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.463350 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.450873    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-envoy\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.491672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.491602    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-token\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.491672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.491603    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-podinfo\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.502327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.502263    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnm9v\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-kube-api-access-gnm9v\") pod \"e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") " pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:28.547882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:28.547458    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:29.022801 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.022756    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"]
Oct 30 22:17:29.024517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.024475    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:29.024629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.024566    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:29.024629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.024605    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:29.207497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.207459    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:29.430758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.430700    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:29.492896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.492859    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" event={"ID":"d35b2afc-b437-4e81-ad4d-4badc5ab4179","Type":"ContainerStarted","Data":"98155c44653c6da379b3365178c291c2e13c58da9fa3e50c0644465eaf6dac37"}
Oct 30 22:17:29.493084 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.492956    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" event={"ID":"d35b2afc-b437-4e81-ad4d-4badc5ab4179","Type":"ContainerStarted","Data":"de2bfba880106b1b8820567cd62855d805a4b089257de4ecd4af4753f4125121"}
Oct 30 22:17:29.549261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.549198    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:29.621203 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:29.621145    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" podStartSLOduration=1.6211293690000002 podStartE2EDuration="1.621129369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:29.617236482 +0000 UTC m=+1661.141379258" watchObservedRunningTime="2026-10-30 22:17:29.621129369 +0000 UTC m=+1661.145272134"
Oct 30 22:17:30.555341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:30.555305    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:31.518786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:31.518749    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:31.519963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:31.519939    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:17:32.467497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:32.467449    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:32.770918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:32.770826    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:33.328551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:33.328516    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:17:33.935627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:33.935584    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:34.335183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:34.335104    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:17:35.892544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:35.888087    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:36.122277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.122238    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:36.203763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.203671    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"]
Oct 30 22:17:36.210138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.210113    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.271264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.271195    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"]
Oct 30 22:17:36.290202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.290165    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1cf0e4-a200-4525-aea5-ab968f265226-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.290418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.290400    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwnq\" (UniqueName: \"kubernetes.io/projected/7d1cf0e4-a200-4525-aea5-ab968f265226-kube-api-access-2dwnq\") pod \"payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.315108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.315064    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-512caa\""
Oct 30 22:17:36.315108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.315075    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-512caa-dockercfg-hjjp5\""
Oct 30 22:17:36.417498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.417459    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwnq\" (UniqueName: \"kubernetes.io/projected/7d1cf0e4-a200-4525-aea5-ab968f265226-kube-api-access-2dwnq\") pod \"payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.428234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.418335    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1cf0e4-a200-4525-aea5-ab968f265226-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.428840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.425746    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"]
Oct 30 22:17:36.454311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.426667    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1cf0e4-a200-4525-aea5-ab968f265226-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.468237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.458526    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:36.531237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.519821    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"]
Oct 30 22:17:36.534661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.534626    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lll\" (UniqueName: \"kubernetes.io/projected/182013cc-390b-477e-8a92-08b67bf64ab8-kube-api-access-t8lll\") pod \"payload-processing-e2e-api-a-512caa-6946957b9-9zspt\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") " pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:36.544165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.543462    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182013cc-390b-477e-8a92-08b67bf64ab8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-512caa-6946957b9-9zspt\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") " pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:36.561471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.558091    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwnq\" (UniqueName: \"kubernetes.io/projected/7d1cf0e4-a200-4525-aea5-ab968f265226-kube-api-access-2dwnq\") pod \"payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.645502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.645471    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lll\" (UniqueName: \"kubernetes.io/projected/182013cc-390b-477e-8a92-08b67bf64ab8-kube-api-access-t8lll\") pod \"payload-processing-e2e-api-a-512caa-6946957b9-9zspt\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") " pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:36.645733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.645530    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182013cc-390b-477e-8a92-08b67bf64ab8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-512caa-6946957b9-9zspt\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") " pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:36.646221 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.646183    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182013cc-390b-477e-8a92-08b67bf64ab8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-512caa-6946957b9-9zspt\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") " pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:36.709512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.709436    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lll\" (UniqueName: \"kubernetes.io/projected/182013cc-390b-477e-8a92-08b67bf64ab8-kube-api-access-t8lll\") pod \"payload-processing-e2e-api-a-512caa-6946957b9-9zspt\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") " pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:36.836422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.836382    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:36.905931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:36.905908    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:37.235859 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:37.235829    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1cf0e4_a200_4525_aea5_ab968f265226.slice/crio-094e9a031ddc6aff6ec6cbedf67488650d11b196f727777b795e376f36e83267 WatchSource:0}: Error finding container 094e9a031ddc6aff6ec6cbedf67488650d11b196f727777b795e376f36e83267: Status 404 returned error can't find the container with id 094e9a031ddc6aff6ec6cbedf67488650d11b196f727777b795e376f36e83267
Oct 30 22:17:37.237529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.237497    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"]
Oct 30 22:17:37.329481 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:37.329450    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182013cc_390b_477e_8a92_08b67bf64ab8.slice/crio-7cedb3770e3ee3ed595749cbb6cf30596ebbbbdd05a83255c996bbf36300b4bb WatchSource:0}: Error finding container 7cedb3770e3ee3ed595749cbb6cf30596ebbbbdd05a83255c996bbf36300b4bb: Status 404 returned error can't find the container with id 7cedb3770e3ee3ed595749cbb6cf30596ebbbbdd05a83255c996bbf36300b4bb
Oct 30 22:17:37.329704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.329675    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"]
Oct 30 22:17:37.617247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.617186    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt" event={"ID":"182013cc-390b-477e-8a92-08b67bf64ab8","Type":"ContainerStarted","Data":"9ff74aac9a873785062a27cad3b8ad8272f56a308e575b3978c58ebc279343a8"}
Oct 30 22:17:37.617476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.617286    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:37.617476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.617311    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt" event={"ID":"182013cc-390b-477e-8a92-08b67bf64ab8","Type":"ContainerStarted","Data":"7cedb3770e3ee3ed595749cbb6cf30596ebbbbdd05a83255c996bbf36300b4bb"}
Oct 30 22:17:37.619237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.619191    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv" event={"ID":"7d1cf0e4-a200-4525-aea5-ab968f265226","Type":"ContainerStarted","Data":"bba9c8aa2147e278695fac69ecd433e8a52653ab125525fec4d79a8d1b9cb5a7"}
Oct 30 22:17:37.619384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.619262    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv" event={"ID":"7d1cf0e4-a200-4525-aea5-ab968f265226","Type":"ContainerStarted","Data":"094e9a031ddc6aff6ec6cbedf67488650d11b196f727777b795e376f36e83267"}
Oct 30 22:17:37.619384 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.619287    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:37.741581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.741523    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt" podStartSLOduration=1.741508524 podStartE2EDuration="1.741508524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:37.738414405 +0000 UTC m=+1669.262557186" watchObservedRunningTime="2026-10-30 22:17:37.741508524 +0000 UTC m=+1669.265651284"
Oct 30 22:17:37.861077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:37.860995    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv" podStartSLOduration=1.86097614 podStartE2EDuration="1.86097614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:37.859857702 +0000 UTC m=+1669.384000468" watchObservedRunningTime="2026-10-30 22:17:37.86097614 +0000 UTC m=+1669.385118907"
Oct 30 22:17:38.482495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:38.482463    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:17:38.482968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:38.482549    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:17:39.522396 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:39.522355    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:44.057526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:44.057491    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"]
Oct 30 22:17:44.107934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:44.107892    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44278f96-jxdrv"]
Oct 30 22:17:44.651436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:44.651395    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f1492e-8155-4281-9ff9-c1ee17fb0b34" path="/var/lib/kubelet/pods/a6f1492e-8155-4281-9ff9-c1ee17fb0b34/volumes"
Oct 30 22:17:44.724493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:44.724455    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:48.412820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:48.412783    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:48.636324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:48.636290    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:17:48.636513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:48.636357    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:17:49.736726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:49.736683    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:50.134798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.134764    5031 scope.go:122] "RemoveContainer" containerID="daa7a7a73de46384f3e21a1a954d3e5a9d43e7eb6155647eb63a50fef6332554"
Oct 30 22:17:50.147588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.147563    5031 scope.go:122] "RemoveContainer" containerID="6dda5b9423de84102e4518ea9b76f413e2a99fa7fca8878321e1128f7be0c00a"
Oct 30 22:17:50.159747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.159718    5031 scope.go:122] "RemoveContainer" containerID="961d1ce8a63ffde351fb3b58ae19cb18f0a163632248bbe4d17d33ff04916e15"
Oct 30 22:17:50.170301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.170280    5031 scope.go:122] "RemoveContainer" containerID="2b4b6fdac4db7b7571a5af4c2fcbf2e10c8375e181c78a62ee050a663c746289"
Oct 30 22:17:50.181412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.181391    5031 scope.go:122] "RemoveContainer" containerID="5054518d1befba2cacc0a7c7d74422d233c5c583d9f738b3c64db0381e6852c9"
Oct 30 22:17:50.191865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.191847    5031 scope.go:122] "RemoveContainer" containerID="41ad170d3a5df11b64084bad3c843df8e665b35569f703939c1fd370d6c491e6"
Oct 30 22:17:50.202525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.202505    5031 scope.go:122] "RemoveContainer" containerID="ab0f45993c261bf67d006db13670c634e417cb21992264d84307bce05a0f10f0"
Oct 30 22:17:50.213237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.213198    5031 scope.go:122] "RemoveContainer" containerID="7719430ea982ad531d879b469a575ead267cf89296fba050cd64adde15bb772e"
Oct 30 22:17:50.223842 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.223819    5031 scope.go:122] "RemoveContainer" containerID="bc9e4eac82daff1894d1e4e297e2e758d64e8e313a88a568b805327fe32ab645"
Oct 30 22:17:50.237032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.237012    5031 scope.go:122] "RemoveContainer" containerID="5c016a0716fb16e6beceb30b1d5a20827fa0d2a23f1df3dda83d23ba523d4ec8"
Oct 30 22:17:50.247711 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:50.247693    5031 scope.go:122] "RemoveContainer" containerID="2ea0c7d34943133100f46063ec03da5ff4de129f54417c79017d804f6b6aa404"
Oct 30 22:17:50.544579 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:17:50.544538    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:17:54.967258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:54.967203    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:55.386562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:55.386520    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:56.170636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.170603    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"]
Oct 30 22:17:56.178908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.178884    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.245134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.245085    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"]
Oct 30 22:17:56.259503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.259474    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-token\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.259687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.259629    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-data\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.259743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.259706    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lkj\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-kube-api-access-b8lkj\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.259801 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.259780    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-envoy\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.259857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.259814    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-credential-socket\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.259857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.259839    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-podinfo\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.260097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.260073    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-socket\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.260198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.260132    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-certs\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.260649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.260622    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istiod-ca-cert\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.333235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.332573    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-9b614c-openshift-default-dockercfg-mcj6q\""
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.352906    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363433    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-data\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363514    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lkj\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-kube-api-access-b8lkj\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363599    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-envoy\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363631    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-credential-socket\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363655    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-podinfo\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363742    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-socket\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363802    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-certs\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363877    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istiod-ca-cert\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.363952    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-token\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.364791    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-data\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.365773    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-socket\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.366101    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-credential-socket\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.367435    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istiod-ca-cert\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.369242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.367793    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-certs\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.375935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.375867    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-envoy\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.379387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.376811    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-podinfo\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.433730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.433636    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-token\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.433730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.433678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lkj\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-kube-api-access-b8lkj\") pod \"e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") " pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.495733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.495699    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:56.574282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:56.574163    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:57.022749 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:17:57.022713    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb3203b_ff2d_4c68_afa6_e19c232d70a0.slice/crio-0e6b0699801562d9ffd5d9252efe5e72a299e7a9186dcd8c8ee3b3f6045c65c2 WatchSource:0}: Error finding container 0e6b0699801562d9ffd5d9252efe5e72a299e7a9186dcd8c8ee3b3f6045c65c2: Status 404 returned error can't find the container with id 0e6b0699801562d9ffd5d9252efe5e72a299e7a9186dcd8c8ee3b3f6045c65c2
Oct 30 22:17:57.023882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:57.023845    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"]
Oct 30 22:17:57.026665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:57.026621    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:57.026774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:57.026717    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:57.026774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:57.026762    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:57.846737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:57.846693    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" event={"ID":"7cb3203b-ff2d-4c68-afa6-e19c232d70a0","Type":"ContainerStarted","Data":"492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23"}
Oct 30 22:17:57.847287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:57.846763    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" event={"ID":"7cb3203b-ff2d-4c68-afa6-e19c232d70a0","Type":"ContainerStarted","Data":"0e6b0699801562d9ffd5d9252efe5e72a299e7a9186dcd8c8ee3b3f6045c65c2"}
Oct 30 22:17:58.496781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:58.496748    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:58.503740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:58.503711    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:58.628807 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:58.628740    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podStartSLOduration=2.628720366 podStartE2EDuration="2.628720366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:58.07769711 +0000 UTC m=+1689.601839873" watchObservedRunningTime="2026-10-30 22:17:58.628720366 +0000 UTC m=+1690.152863132"
Oct 30 22:17:58.859688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:58.859659    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:58.860650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:58.860622    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:17:59.714959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:17:59.714921    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:18:00.397388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:00.397344    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:01.038806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:01.038772    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:01.310999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:01.310916    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:04.450910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:04.450872    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:04.787917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:04.787787    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:06.608524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:06.608491    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:11.070022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:11.069981    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:11.546570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:11.546528    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:12.055342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.055308    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"]
Oct 30 22:18:12.060473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.060448    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165414    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-data\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165482    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-credential-socket\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165575    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjgg\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-kube-api-access-mkjgg\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165633    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-token\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165708    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-socket\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165772    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-certs\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165838    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istiod-ca-cert\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165879    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-envoy\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.167162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.165928    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-podinfo\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.175091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.175063    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c3d917-openshift-default-dockercfg-pfgr2\""
Oct 30 22:18:12.241235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.240652    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:12.306300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.302964    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-data\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.306300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.303023    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-credential-socket\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.306300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.303115    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjgg\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-kube-api-access-mkjgg\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.306300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.303171    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-token\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.306300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.303268    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-socket\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.306300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.303333    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-certs\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.306300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.303398    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istiod-ca-cert\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.349305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.330266    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-data\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.349305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.330574    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-credential-socket\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.349305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.330829    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-socket\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.349305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.331373    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-certs\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.361508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.354983    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istiod-ca-cert\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.361508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.358763    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-envoy\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.361508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.358841    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-podinfo\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.369522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.368780    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"]
Oct 30 22:18:12.369522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.369021    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-envoy\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.369760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.369605    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-podinfo\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.398176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.398145    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-token\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.400436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.400404    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjgg\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-kube-api-access-mkjgg\") pod \"e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") " pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.540294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.535347    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:12.685319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.685282    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:12.853241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:12.851581    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:13.155721 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:13.155692    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"]
Oct 30 22:18:13.156775 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:18:13.156740    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609f40a6_6a81_4eaa_b191_e414cf8bc4cf.slice/crio-f4795ca2d93fbdd268589cc8ff6035a504cd2b9117b81115547df6524f631ea4 WatchSource:0}: Error finding container f4795ca2d93fbdd268589cc8ff6035a504cd2b9117b81115547df6524f631ea4: Status 404 returned error can't find the container with id f4795ca2d93fbdd268589cc8ff6035a504cd2b9117b81115547df6524f631ea4
Oct 30 22:18:13.159939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:13.159903    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:13.160029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:13.159981    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:13.160029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:13.160007    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:14.011956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:14.011916    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" event={"ID":"609f40a6-6a81-4eaa-b191-e414cf8bc4cf","Type":"ContainerStarted","Data":"0892340848cbd133c527e638e31ba9597ece28bf3c9c7effc1adc68505c5b06f"}
Oct 30 22:18:14.012513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:14.011975    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" event={"ID":"609f40a6-6a81-4eaa-b191-e414cf8bc4cf","Type":"ContainerStarted","Data":"f4795ca2d93fbdd268589cc8ff6035a504cd2b9117b81115547df6524f631ea4"}
Oct 30 22:18:14.184625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:14.184552    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podStartSLOduration=2.184537129 podStartE2EDuration="2.184537129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:14.180556492 +0000 UTC m=+1705.704699281" watchObservedRunningTime="2026-10-30 22:18:14.184537129 +0000 UTC m=+1705.708679970"
Oct 30 22:18:14.685433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:14.685397    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:14.692149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:14.692122    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:15.023279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:15.023147    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:15.024349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:15.024324    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:18:17.072061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:17.072028    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:17.345832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:17.345799    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.103546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:18.103482    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.340762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:18.340695    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.601758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:18.601727    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:20.455706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:20.455662    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:20.547808 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:18:20.547761    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:18:20.758796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:20.758694    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:21.182449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.182418    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"]
Oct 30 22:18:21.195237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.195151    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.202254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.202181    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"]
Oct 30 22:18:21.281296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.281254    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7787089a-e528-4c5c-b083-dc3d94fb6db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.281500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.281401    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8pbc\" (UniqueName: \"kubernetes.io/projected/7787089a-e528-4c5c-b083-dc3d94fb6db4-kube-api-access-p8pbc\") pod \"payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.319538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.319502    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c3d917-dockercfg-mglgr\""
Oct 30 22:18:21.319538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.319502    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c3d917\""
Oct 30 22:18:21.382544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.382511    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8pbc\" (UniqueName: \"kubernetes.io/projected/7787089a-e528-4c5c-b083-dc3d94fb6db4-kube-api-access-p8pbc\") pod \"payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.382769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.382748    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7787089a-e528-4c5c-b083-dc3d94fb6db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.383758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.383729    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7787089a-e528-4c5c-b083-dc3d94fb6db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.447730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.447648    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8pbc\" (UniqueName: \"kubernetes.io/projected/7787089a-e528-4c5c-b083-dc3d94fb6db4-kube-api-access-p8pbc\") pod \"payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.527243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.524271    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:21.945052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:21.945026    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"]
Oct 30 22:18:21.946358 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:18:21.946327    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7787089a_e528_4c5c_b083_dc3d94fb6db4.slice/crio-22f8dac918f3603e75e22ef7ecad5a73ca392d0ca0e1b6013a74603f638b5029 WatchSource:0}: Error finding container 22f8dac918f3603e75e22ef7ecad5a73ca392d0ca0e1b6013a74603f638b5029: Status 404 returned error can't find the container with id 22f8dac918f3603e75e22ef7ecad5a73ca392d0ca0e1b6013a74603f638b5029
Oct 30 22:18:22.190662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:22.190624    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4" event={"ID":"7787089a-e528-4c5c-b083-dc3d94fb6db4","Type":"ContainerStarted","Data":"ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958"}
Oct 30 22:18:22.190854 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:22.190676    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4" event={"ID":"7787089a-e528-4c5c-b083-dc3d94fb6db4","Type":"ContainerStarted","Data":"22f8dac918f3603e75e22ef7ecad5a73ca392d0ca0e1b6013a74603f638b5029"}
Oct 30 22:18:22.190854 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:22.190719    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:22.317299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:22.317176    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4" podStartSLOduration=1.317160904 podStartE2EDuration="1.317160904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:22.312314545 +0000 UTC m=+1713.836457311" watchObservedRunningTime="2026-10-30 22:18:22.317160904 +0000 UTC m=+1713.841303667"
Oct 30 22:18:24.196399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:24.196362    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.532219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:25.532172    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:29.658637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:29.658588    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:29.907321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:29.907274    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:33.087244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:33.084970    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"]
Oct 30 22:18:33.130825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:33.130772    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a8cb80-2xfnp"]
Oct 30 22:18:33.204184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:33.204154    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:18:34.661644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:34.661598    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe45f2c-e313-4a02-a850-83d833003308" path="/var/lib/kubelet/pods/cfe45f2c-e313-4a02-a850-83d833003308/volumes"
Oct 30 22:18:35.600421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:35.600366    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:41.087747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:41.087653    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:42.475493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:42.475453    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:46.569001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:46.568958    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:46.984000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:46.983962    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:47.205506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:47.205454    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:18:47.327595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:47.327496    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:48.831398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:48.831342    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:50.447581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.447544    5031 scope.go:122] "RemoveContainer" containerID="3d26ccdab8622074eb72c2490f6ccbceae958a533ecac72976ea07516844a575"
Oct 30 22:18:50.460347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.460326    5031 scope.go:122] "RemoveContainer" containerID="3db38be749b3e90ebc4abbb42c7b7a0cd7dfc2550e1e38b4c67e1cb72ff0de35"
Oct 30 22:18:50.495908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.473828    5031 scope.go:122] "RemoveContainer" containerID="dbc00935814b0cb7cbdfa9e9e809ecd2ff209e22eb1f433d85b4a3b5a1d55624"
Oct 30 22:18:50.681225 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:18:50.681179    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:18:50.739956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.739843    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"]
Oct 30 22:18:50.746388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.746359    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.788581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.788539    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"]
Oct 30 22:18:50.838292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.837582    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:18:50.838292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.837594    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:18:50.838292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.837742    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:50.853228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.851268    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:18:50.884787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.884744    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmt5m\" (UniqueName: \"kubernetes.io/projected/a0b32122-90c6-48ab-83ef-248273951781-kube-api-access-nmt5m\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.885015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.884869    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b32122-90c6-48ab-83ef-248273951781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.885083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.885014    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b32122-90c6-48ab-83ef-248273951781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.986588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.986552    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmt5m\" (UniqueName: \"kubernetes.io/projected/a0b32122-90c6-48ab-83ef-248273951781-kube-api-access-nmt5m\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.986788 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.986658    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b32122-90c6-48ab-83ef-248273951781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.986858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.986791    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b32122-90c6-48ab-83ef-248273951781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.987560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.987532    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b32122-90c6-48ab-83ef-248273951781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:50.989440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:50.989422    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b32122-90c6-48ab-83ef-248273951781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:51.054847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:51.054762    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmt5m\" (UniqueName: \"kubernetes.io/projected/a0b32122-90c6-48ab-83ef-248273951781-kube-api-access-nmt5m\") pod \"maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:51.063492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:51.063465    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:51.497700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:51.497664    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"]
Oct 30 22:18:51.498164 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:18:51.498116    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b32122_90c6_48ab_83ef_248273951781.slice/crio-10524b517366cb68bb2b24beb148c6d0f5db26771e37bc2c9e4baa936950beac WatchSource:0}: Error finding container 10524b517366cb68bb2b24beb148c6d0f5db26771e37bc2c9e4baa936950beac: Status 404 returned error can't find the container with id 10524b517366cb68bb2b24beb148c6d0f5db26771e37bc2c9e4baa936950beac
Oct 30 22:18:51.556831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:51.556798    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67" event={"ID":"a0b32122-90c6-48ab-83ef-248273951781","Type":"ContainerStarted","Data":"10524b517366cb68bb2b24beb148c6d0f5db26771e37bc2c9e4baa936950beac"}
Oct 30 22:18:52.569177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:52.569139    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67" event={"ID":"a0b32122-90c6-48ab-83ef-248273951781","Type":"ContainerStarted","Data":"e12e7948309001d1774a37b9c7d01c08e292fc47f3f4906a89475468c9fbbbf1"}
Oct 30 22:18:52.697681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:52.697627    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67" podStartSLOduration=2.697612943 podStartE2EDuration="2.697612943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:52.694253205 +0000 UTC m=+1744.218395967" watchObservedRunningTime="2026-10-30 22:18:52.697612943 +0000 UTC m=+1744.221755760"
Oct 30 22:18:53.081707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:53.081662    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:53.578712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:53.578680    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0b32122-90c6-48ab-83ef-248273951781" containerID="e12e7948309001d1774a37b9c7d01c08e292fc47f3f4906a89475468c9fbbbf1" exitCode=0
Oct 30 22:18:53.579167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:53.578779    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67" event={"ID":"a0b32122-90c6-48ab-83ef-248273951781","Type":"ContainerDied","Data":"e12e7948309001d1774a37b9c7d01c08e292fc47f3f4906a89475468c9fbbbf1"}
Oct 30 22:18:54.800518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.800491    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:54.885193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.885155    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b32122-90c6-48ab-83ef-248273951781-kube-api-access-nmt5m\" (UniqueName: \"kubernetes.io/projected/a0b32122-90c6-48ab-83ef-248273951781-kube-api-access-nmt5m\") pod \"a0b32122-90c6-48ab-83ef-248273951781\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") "
Oct 30 22:18:54.885428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.885305    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b32122-90c6-48ab-83ef-248273951781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b32122-90c6-48ab-83ef-248273951781-maas-api-xks-ca\") pod \"a0b32122-90c6-48ab-83ef-248273951781\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") "
Oct 30 22:18:54.885621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.885600    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b32122-90c6-48ab-83ef-248273951781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b32122-90c6-48ab-83ef-248273951781-maas-api-openshift-ca\") pod \"a0b32122-90c6-48ab-83ef-248273951781\" (UID: \"a0b32122-90c6-48ab-83ef-248273951781\") "
Oct 30 22:18:54.886177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.885949    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b32122-90c6-48ab-83ef-248273951781-maas-api-openshift-ca" pod "a0b32122-90c6-48ab-83ef-248273951781" (UID: "a0b32122-90c6-48ab-83ef-248273951781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:54.886332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.886244    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b32122-90c6-48ab-83ef-248273951781-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:54.887644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.887618    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b32122-90c6-48ab-83ef-248273951781-maas-api-xks-ca" pod "a0b32122-90c6-48ab-83ef-248273951781" (UID: "a0b32122-90c6-48ab-83ef-248273951781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:54.887785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.887693    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b32122-90c6-48ab-83ef-248273951781-kube-api-access-nmt5m" pod "a0b32122-90c6-48ab-83ef-248273951781" (UID: "a0b32122-90c6-48ab-83ef-248273951781"). InnerVolumeSpecName "kube-api-access-nmt5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:54.988411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.988370    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmt5m\" (UniqueName: \"kubernetes.io/projected/a0b32122-90c6-48ab-83ef-248273951781-kube-api-access-nmt5m\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:54.988591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:54.988573    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b32122-90c6-48ab-83ef-248273951781-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.527620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:55.527582    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:55.603074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:55.603041    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"
Oct 30 22:18:55.603314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:55.603045    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67" event={"ID":"a0b32122-90c6-48ab-83ef-248273951781","Type":"ContainerDied","Data":"10524b517366cb68bb2b24beb148c6d0f5db26771e37bc2c9e4baa936950beac"}
Oct 30 22:18:55.603314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:55.603182    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10524b517366cb68bb2b24beb148c6d0f5db26771e37bc2c9e4baa936950beac"
Oct 30 22:18:58.635352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:58.635317    5031 generic.go:356] "Generic (PLEG): container finished" podID="591bc892-852c-416c-9df2-e61a8d9a2015" containerID="5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490" exitCode=0
Oct 30 22:18:58.635814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:58.635393    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" event={"ID":"591bc892-852c-416c-9df2-e61a8d9a2015","Type":"ContainerDied","Data":"5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490"}
Oct 30 22:18:58.635995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:58.635977    5031 scope.go:122] "RemoveContainer" containerID="5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490"
Oct 30 22:18:58.638228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:58.638174    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:58.638635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:58.638288    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:58.638635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:58.638339    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:58.923385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:58.923272    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:59.649695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:18:59.649656    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" event={"ID":"591bc892-852c-416c-9df2-e61a8d9a2015","Type":"ContainerStarted","Data":"d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f"}
Oct 30 22:19:00.098665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:00.098638    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:19:00.125235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:00.114651    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:19:00.125235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:00.114784    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:19:00.125235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:00.114840    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:19:01.130238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:01.119872    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:19:01.130238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:01.119938    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:19:01.573648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:01.573464    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:01.803616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:01.803586    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:02.125749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:02.125709    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:19:02.126117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:02.126076    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:19:03.114254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:03.112523    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:19:03.114254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:03.112598    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:19:04.112677 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:04.112634    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:19:04.113002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:04.112969    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:19:04.342841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:04.342801    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:05.117237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:05.111688    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:19:05.117237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:05.111755    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:19:06.114045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:06.114013    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:19:06.115220 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:06.115181    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:19:09.520153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:09.520117    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:14.648079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:14.648023    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:19.655927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:19.655861    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.545248 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:19:20.545186    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:19:21.468824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:21.468790    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:24.915237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:24.914842    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:30.190335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:30.190295    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:34.578555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:34.578519    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:36.188958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:36.188919    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:36.415843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:36.415805    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:36.640821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:36.640787    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:36.652459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:36.652421    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"]
Oct 30 22:19:36.652638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:36.652475    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7422b4b3-7fp67"]
Oct 30 22:19:37.062795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.062695    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"]
Oct 30 22:19:37.063017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.062917    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" containerID="cri-o://ed7e3058036a129f410de6a728c5cd8326cb52af5852074c27843d0c95f57f13" gracePeriod=30
Oct 30 22:19:37.151201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.151163    5031 generic.go:356] "Generic (PLEG): container finished" podID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerID="98155c44653c6da379b3365178c291c2e13c58da9fa3e50c0644465eaf6dac37" exitCode=0
Oct 30 22:19:37.151201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.151261    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" event={"ID":"d35b2afc-b437-4e81-ad4d-4badc5ab4179","Type":"ContainerDied","Data":"98155c44653c6da379b3365178c291c2e13c58da9fa3e50c0644465eaf6dac37"}
Oct 30 22:19:37.151713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.151694    5031 scope.go:122] "RemoveContainer" containerID="98155c44653c6da379b3365178c291c2e13c58da9fa3e50c0644465eaf6dac37"
Oct 30 22:19:37.153720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.153677    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:37.153872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.153761    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:37.153872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.153808    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:37.230243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:37.230155    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:38.163742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:38.163702    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" event={"ID":"d35b2afc-b437-4e81-ad4d-4badc5ab4179","Type":"ContainerStarted","Data":"e81150e6716343faef43211b7219d6dd044cdbf72b8b1bfba65b699dc6741d00"}
Oct 30 22:19:38.548769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:38.548674    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:19:38.549150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:38.549051    5031 patch_prober.go:29] interesting pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:19:38.549150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:38.549105    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 22:19:38.667648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:38.667612    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b32122-90c6-48ab-83ef-248273951781" path="/var/lib/kubelet/pods/a0b32122-90c6-48ab-83ef-248273951781/volumes"
Oct 30 22:19:38.932421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:38.932386    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:38.932701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:38.932667    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:39.550667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:39.550628    5031 patch_prober.go:29] interesting pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:19:39.551281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:39.551247    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 22:19:40.548751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:40.548709    5031 patch_prober.go:29] interesting pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:19:40.548978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:40.548793    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 22:19:41.317236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:41.306530    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"]
Oct 30 22:19:41.318138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:41.318097    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy" containerID="cri-o://ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63" gracePeriod=30
Oct 30 22:19:41.548370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:41.548332    5031 patch_prober.go:29] interesting pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:19:41.548551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:41.548386    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 22:19:42.028243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.028181    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:42.208818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.208780    5031 generic.go:356] "Generic (PLEG): container finished" podID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerID="ed7e3058036a129f410de6a728c5cd8326cb52af5852074c27843d0c95f57f13" exitCode=0
Oct 30 22:19:42.213235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.209388    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" event={"ID":"0a217c30-ee84-4544-9ff6-d03466b88f7b","Type":"ContainerDied","Data":"ed7e3058036a129f410de6a728c5cd8326cb52af5852074c27843d0c95f57f13"}
Oct 30 22:19:42.442280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.441947    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:19:42.531237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.530733    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-socket\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.530808    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-podinfo\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.530867    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a217c30-ee84-4544-9ff6-d03466b88f7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a217c30-ee84-4544-9ff6-d03466b88f7b-istiod-ca-cert\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.530958    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-token\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-token\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.531008    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-envoy\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.531115    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-credential-socket\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.531288    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-kube-api-access-brmhn\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-kube-api-access-brmhn\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.531630    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a217c30-ee84-4544-9ff6-d03466b88f7b-istiod-ca-cert" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:42.531723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.531693    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-certs\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.531896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.531782    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-data\") pod \"0a217c30-ee84-4544-9ff6-d03466b88f7b\" (UID: \"0a217c30-ee84-4544-9ff6-d03466b88f7b\") "
Oct 30 22:19:42.535928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.532690    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a217c30-ee84-4544-9ff6-d03466b88f7b-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.535928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.532885    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-data" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.535928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.533029    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-socket" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.535928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.535766    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-kube-api-access-brmhn" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "kube-api-access-brmhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:42.536325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.535929    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-certs" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.536325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.536043    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-credential-socket" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.538133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.538105    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-envoy" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.538733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.538708    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-podinfo" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:42.539476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.539455    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-token" pod "0a217c30-ee84-4544-9ff6-d03466b88f7b" (UID: "0a217c30-ee84-4544-9ff6-d03466b88f7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:42.549055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.549034    5031 patch_prober.go:29] interesting pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:19:42.549112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.549087    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638527    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmhn\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-kube-api-access-brmhn\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638569    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638583    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638596    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638609    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638622    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638634    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.639219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:42.638646    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a217c30-ee84-4544-9ff6-d03466b88f7b-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:43.246446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.245800    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5" event={"ID":"0a217c30-ee84-4544-9ff6-d03466b88f7b","Type":"ContainerDied","Data":"ee425a77d6c6b43674508858d8f95486eda6fa22b5b055918e28b0a69f77749a"}
Oct 30 22:19:43.246446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.245875    5031 scope.go:122] "RemoveContainer" containerID="ed7e3058036a129f410de6a728c5cd8326cb52af5852074c27843d0c95f57f13"
Oct 30 22:19:43.246446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.246062    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"
Oct 30 22:19:43.421675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.421639    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"]
Oct 30 22:19:43.472190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.472115    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5"]
Oct 30 22:19:43.552905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.552823    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:19:43.553407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.553381    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:19:43.554088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:43.554068    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:19:44.648749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:44.648709    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" path="/var/lib/kubelet/pods/0a217c30-ee84-4544-9ff6-d03466b88f7b/volumes"
Oct 30 22:19:45.069072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:45.068973    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:45.314130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:45.314096    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:45.567236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:45.558717    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:46.444083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.444044    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:46.893276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.893244    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:19:46.968087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968047    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-certs\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968171    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-podinfo\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968309    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-token\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-token\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968377    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-envoy\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968413    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-kube-api-access-dm5vz\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-kube-api-access-dm5vz\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968466    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-data\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968519    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-credential-socket\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968589    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f709b7-4a42-46f4-b20e-6b6a37933a23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f709b7-4a42-46f4-b20e-6b6a37933a23-istiod-ca-cert\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968652    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-socket\") pod \"32f709b7-4a42-46f4-b20e-6b6a37933a23\" (UID: \"32f709b7-4a42-46f4-b20e-6b6a37933a23\") "
Oct 30 22:19:46.968917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.968886    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-certs" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:46.969377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.969349    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-credential-socket" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:46.969511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.969486    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-data" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:46.969703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.969681    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f709b7-4a42-46f4-b20e-6b6a37933a23-istiod-ca-cert" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:46.969828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.969807    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:46.969945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.969931    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:46.970062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.970049    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:46.970163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.969858    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-socket" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:46.971531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.971495    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-podinfo" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:46.974439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.974413    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-envoy" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:46.976186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.976161    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-kube-api-access-dm5vz" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "kube-api-access-dm5vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:46.976734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:46.976714    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-token" pod "32f709b7-4a42-46f4-b20e-6b6a37933a23" (UID: "32f709b7-4a42-46f4-b20e-6b6a37933a23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:47.145773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.113806    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f709b7-4a42-46f4-b20e-6b6a37933a23-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.145773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.113845    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.145773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.113858    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.145773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.113869    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.145773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.113881    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f709b7-4a42-46f4-b20e-6b6a37933a23-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.145773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.113893    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5vz\" (UniqueName: \"kubernetes.io/projected/32f709b7-4a42-46f4-b20e-6b6a37933a23-kube-api-access-dm5vz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.222865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.220492    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:47.361046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.359578    5031 generic.go:356] "Generic (PLEG): container finished" podID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerID="ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63" exitCode=0
Oct 30 22:19:47.361046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.359897    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" event={"ID":"32f709b7-4a42-46f4-b20e-6b6a37933a23","Type":"ContainerDied","Data":"ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63"}
Oct 30 22:19:47.361046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.359959    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq" event={"ID":"32f709b7-4a42-46f4-b20e-6b6a37933a23","Type":"ContainerDied","Data":"398c4d8985cecc1f1ba1149565abe13b0af86be9d6809f503f70e470ce40ee23"}
Oct 30 22:19:47.361046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.359994    5031 scope.go:122] "RemoveContainer" containerID="ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63"
Oct 30 22:19:47.361046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.360164    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"
Oct 30 22:19:47.382606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.382305    5031 scope.go:122] "RemoveContainer" containerID="ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63"
Oct 30 22:19:47.382706 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:19:47.382659    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63\": container with ID starting with ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63 not found: ID does not exist" containerID="ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63"
Oct 30 22:19:47.382769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.382714    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63"} err="failed to get container status \"ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63\": rpc error: code = NotFound desc = could not find container \"ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63\": container with ID starting with ac8ddecc08d7d9bc066ab57aed2758aac83ce52d52563e6f9871327a2a117d63 not found: ID does not exist"
Oct 30 22:19:47.496759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.496668    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:47.539452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.539414    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"]
Oct 30 22:19:47.559488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.559440    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq"]
Oct 30 22:19:47.750666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:47.750568    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:48.647939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:48.647904    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" path="/var/lib/kubelet/pods/32f709b7-4a42-46f4-b20e-6b6a37933a23/volumes"
Oct 30 22:19:49.107867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:49.107833    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:49.445894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:49.445765    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:49.636261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:49.636228    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:19:49.659127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:49.659101    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:19:49.913577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:49.912748    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:50.369921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:50.369886    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:50.550961 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:19:50.550919    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:19:50.697993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:50.697885    5031 scope.go:122] "RemoveContainer" containerID="329a7fdf468dfd0270504ab7f3b1222f90f2035d1aa01c4fea7f2f09e4dc905c"
Oct 30 22:19:50.720244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:50.720001    5031 scope.go:122] "RemoveContainer" containerID="11c857ad1d43bd4c03933f8fc7450c64b80b714fe33907c0697b8d832fa442e5"
Oct 30 22:19:50.736121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:50.736095    5031 scope.go:122] "RemoveContainer" containerID="018d59d88e1c5d811e79af9cf9ff2153be4c8de162dac3283fb3a4597d60c1db"
Oct 30 22:19:50.763406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:50.763377    5031 scope.go:122] "RemoveContainer" containerID="e003e3907f35f3f8bde71dd106ad899aca448f4e8682cba275bea383c790f177"
Oct 30 22:19:51.353234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:51.349501    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:51.583723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:51.583680    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:52.256988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:52.256949    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:52.502766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:52.502732    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:53.171633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:53.171603    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:55.449234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:55.448321    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:55.726298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:55.722499    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:19:55.729776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:55.728918    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:57.857959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:57.857917    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:58.086259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:58.086172    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:59.127238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:19:59.123943    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:01.089148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:01.089110    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:01.383830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:01.383785    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:01.672912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:01.672796    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:02.535913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:02.535857    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:03.178464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:03.178412    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:04.787527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:04.787326    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:06.240789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:06.240386    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:20:06.240789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:06.240483    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:07.227874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:07.227819    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:07.468165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:07.468131    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:09.869076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:09.869043    5031 generic.go:356] "Generic (PLEG): container finished" podID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerID="492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23" exitCode=0
Oct 30 22:20:09.870360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:09.869118    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" event={"ID":"7cb3203b-ff2d-4c68-afa6-e19c232d70a0","Type":"ContainerDied","Data":"492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23"}
Oct 30 22:20:09.870360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:09.869627    5031 scope.go:122] "RemoveContainer" containerID="492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23"
Oct 30 22:20:09.871624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:09.871595    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:09.871692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:09.871667    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:09.871729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:09.871700    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:10.405122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.405071    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:10.701579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.701425    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:10.896957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.896922    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" event={"ID":"7cb3203b-ff2d-4c68-afa6-e19c232d70a0","Type":"ContainerStarted","Data":"9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13"}
Oct 30 22:20:10.903283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.902883    5031 generic.go:356] "Generic (PLEG): container finished" podID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerID="c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2" exitCode=0
Oct 30 22:20:10.903283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.902928    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" event={"ID":"895bc537-7f1c-4950-bfa0-b89ae9531f95","Type":"ContainerDied","Data":"c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2"}
Oct 30 22:20:10.904281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.903792    5031 scope.go:122] "RemoveContainer" containerID="c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2"
Oct 30 22:20:10.908480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.908439    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:10.908639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.908543    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:10.908639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.908594    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:10.956128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:10.956024    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:11.496814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:11.496784    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:20:11.498134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:11.498106    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.498402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:11.498372    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 22:20:11.950717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:11.950675    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" event={"ID":"895bc537-7f1c-4950-bfa0-b89ae9531f95","Type":"ContainerStarted","Data":"f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6"}
Oct 30 22:20:12.003219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:12.003183    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:20:12.003465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:12.003442    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.003516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:12.003494    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:20:12.497069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:12.497034    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.497285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:12.497103    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 22:20:13.003943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:13.003634    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.003943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:13.003696    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:20:13.497009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:13.496933    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.497260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:13.497036    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 22:20:13.872234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:13.871311    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:20:14.003626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:14.003591    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.003832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:14.003648    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:20:14.497094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:14.496647    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.497094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:14.496713    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 22:20:15.004028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:15.003991    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:20:15.004268 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:15.004048    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:20:15.529248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:15.526886    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:20:15.529248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:15.526979    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 22:20:16.004066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:16.004033    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:20:16.004287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:16.004089    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:20:16.131232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:16.126120    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:16.497145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:16.497103    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:20:16.497355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:16.497165    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 22:20:17.003628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:17.003593    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:20:17.004058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:17.003645    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:20:17.502328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:17.502289    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:20:17.503544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:17.503519    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:20:18.003834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:18.003788    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:20:18.004332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:18.003862    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:20:19.009267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:19.009229    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:20:19.009678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:19.009645    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:20:19.010579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:19.010555    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:20:20.544474 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:20:20.544432    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:20:21.613903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:21.613865    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:21.870907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:21.870826    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:22.432653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:22.432624    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:23.541941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:23.541901    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:23.864907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:23.864868    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:26.914860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:26.914826    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:27.542327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:27.542196    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:27.774894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:27.774859    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:30.930827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:30.930789    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:31.198508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:31.198420    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.660666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:32.660624    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:38.167270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:38.160744    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:41.553583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:41.553524    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:20:43.562904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:43.562866    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:48.947403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:48.947369    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:49.482594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:49.482558    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:49.981479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:49.981443    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:50.423841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.423803    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:50.552141 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:20:50.552100    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:20:50.871603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.871562    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"]
Oct 30 22:20:50.872436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872413    5031 cpu_manager.go:414] "containerMap: removing container" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy"
Oct 30 22:20:50.872436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872438    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy"
Oct 30 22:20:50.872601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872492    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0b32122-90c6-48ab-83ef-248273951781" containerName="revoke-keys"
Oct 30 22:20:50.872601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872498    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b32122-90c6-48ab-83ef-248273951781" containerName="revoke-keys"
Oct 30 22:20:50.872601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872507    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy"
Oct 30 22:20:50.872601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872512    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy"
Oct 30 22:20:50.872726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872650    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f709b7-4a42-46f4-b20e-6b6a37933a23" containerName="istio-proxy"
Oct 30 22:20:50.872726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872665    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b32122-90c6-48ab-83ef-248273951781" containerName="revoke-keys"
Oct 30 22:20:50.872726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.872672    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a217c30-ee84-4544-9ff6-d03466b88f7b" containerName="istio-proxy"
Oct 30 22:20:50.901426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.901391    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.017259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:50.958923    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7cfd50-openshift-default-dockercfg-9zj2f\""
Oct 30 22:20:51.120587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.120553    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:51.158385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.158296    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"]
Oct 30 22:20:51.178542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178287    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istiod-ca-cert\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178359    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-certs\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178400    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72t2g\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-kube-api-access-72t2g\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178492    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-socket\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178562    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-envoy\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178588    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-podinfo\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178668    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-data\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178716    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-credential-socket\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.178927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.178920    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-token\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281089    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istiod-ca-cert\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-certs\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281193    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72t2g\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-kube-api-access-72t2g\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281326    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-socket\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281391    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-envoy\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281419    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-podinfo\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281491    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-data\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281539    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-credential-socket\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281725    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-token\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281752    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-certs\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.281904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.281832    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-socket\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.282098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.282072    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-credential-socket\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.282844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.282817    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istiod-ca-cert\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.282995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.282941    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-data\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.284478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.284447    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-envoy\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.284478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.284450    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-podinfo\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.347659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.347630    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-token\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.348405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.348385    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72t2g\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-kube-api-access-72t2g\") pod \"e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") " pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.555443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.555343    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:51.721399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:51.721193    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.075345 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:20:52.075314    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fcb1e3_b93e_42e8_b489_bbec1d3409de.slice/crio-1bd66b7898fe4f234c13871eb9707f5f089676d04fb99dbaafcce2c9355c1fd9 WatchSource:0}: Error finding container 1bd66b7898fe4f234c13871eb9707f5f089676d04fb99dbaafcce2c9355c1fd9: Status 404 returned error can't find the container with id 1bd66b7898fe4f234c13871eb9707f5f089676d04fb99dbaafcce2c9355c1fd9
Oct 30 22:20:52.077148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.077122    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"]
Oct 30 22:20:52.078828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.078797    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:52.078895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.078862    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:52.078933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.078893    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:52.459194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.459152    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" event={"ID":"f6fcb1e3-b93e-42e8-b489-bbec1d3409de","Type":"ContainerStarted","Data":"b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd"}
Oct 30 22:20:52.459427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.459251    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" event={"ID":"f6fcb1e3-b93e-42e8-b489-bbec1d3409de","Type":"ContainerStarted","Data":"1bd66b7898fe4f234c13871eb9707f5f089676d04fb99dbaafcce2c9355c1fd9"}
Oct 30 22:20:52.556197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.556156    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:52.585354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:52.585287    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" podStartSLOduration=2.585271717 podStartE2EDuration="2.585271717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:52.584407075 +0000 UTC m=+1864.108549866" watchObservedRunningTime="2026-10-30 22:20:52.585271717 +0000 UTC m=+1864.109414481"
Oct 30 22:20:53.477507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:53.477455    5031 generic.go:356] "Generic (PLEG): container finished" podID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerID="0892340848cbd133c527e638e31ba9597ece28bf3c9c7effc1adc68505c5b06f" exitCode=0
Oct 30 22:20:53.478160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:53.477527    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" event={"ID":"609f40a6-6a81-4eaa-b191-e414cf8bc4cf","Type":"ContainerDied","Data":"0892340848cbd133c527e638e31ba9597ece28bf3c9c7effc1adc68505c5b06f"}
Oct 30 22:20:53.478367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:53.478339    5031 scope.go:122] "RemoveContainer" containerID="0892340848cbd133c527e638e31ba9597ece28bf3c9c7effc1adc68505c5b06f"
Oct 30 22:20:53.480201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:53.480167    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:53.480325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:53.480285    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:53.480325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:53.480317    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:53.562841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:53.562811    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:54.336201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:54.336161    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:54.499143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:54.493677    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" event={"ID":"609f40a6-6a81-4eaa-b191-e414cf8bc4cf","Type":"ContainerStarted","Data":"238a5ecfc07f41066ffbf6085df1737a707abeb9460391f2662e4bf87576a543"}
Oct 30 22:20:54.499143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:54.493803    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:54.499143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:54.495348    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:20:54.686292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:54.686257    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:20:54.686534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:54.686516    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:20:54.686582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:54.686558    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 22:20:55.647651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:55.647456    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:55.686389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:55.686354    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:20:55.686579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:55.686407    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 22:20:55.876636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:55.876598    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:56.686486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:56.686450    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:20:56.686911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:56.686512    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 22:20:57.686235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:57.686168    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:20:57.686434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:57.686240    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 22:20:58.686152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:58.686104    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:20:58.686630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:58.686196    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 22:20:59.685911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:59.685871    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:20:59.686123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:20:59.685940    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 22:21:00.024113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:00.024022    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:21:00.082646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:00.082593    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:00.690839 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:00.690805    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:21:00.692104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:00.692077    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:21:00.702620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:00.702566    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:00.941448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:00.941349    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:05.639517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:05.639474    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:10.828446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:10.828404    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:11.274423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:11.274388    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.473332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:12.473296    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:13.179292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:13.172809    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:13.621953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:13.621914    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:14.801371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:14.801335    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:15.787171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:15.787098    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:16.025706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:16.025674    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:16.401547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:16.401511    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:17.885804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:17.885772    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:18.197967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:18.197858    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:20.544986 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:21:20.544949    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:21:21.189726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:21.189678    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:21.433660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:21.433609    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.355513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:26.355478    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.596674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:26.596640    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:27.986100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:27.986064    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:28.387503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:28.387464    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:31.726562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:31.726476    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:36.766102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:36.766053    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:41.871849 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:41.871750    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:42.102772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.102735    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:42.276520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.276422    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"]
Oct 30 22:21:42.279923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.279898    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.340360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.340324    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"]
Oct 30 22:21:42.357087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.357051    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnnr\" (UniqueName: \"kubernetes.io/projected/d802ee16-0512-4053-a1e2-d12ab06210f9-kube-api-access-gsnnr\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.357318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.357117    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.357601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.357571    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.384905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.384820    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:42.384905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.384830    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:42.385195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.384925    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:21:42.385195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.384925    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:42.459679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.459639    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.459918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.459705    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnnr\" (UniqueName: \"kubernetes.io/projected/d802ee16-0512-4053-a1e2-d12ab06210f9-kube-api-access-gsnnr\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.459918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.459749    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.460472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.460443    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.462301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.462278    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.523792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.523750    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnnr\" (UniqueName: \"kubernetes.io/projected/d802ee16-0512-4053-a1e2-d12ab06210f9-kube-api-access-gsnnr\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-px7cr\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.593620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.593585    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:42.992755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:42.992724    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"]
Oct 30 22:21:42.993187 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:21:42.993137    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd802ee16_0512_4053_a1e2_d12ab06210f9.slice/crio-3a8ae512781e6df141b2b5fe20c064d45c5cba17551ed2a4ffabd6bf493a354f WatchSource:0}: Error finding container 3a8ae512781e6df141b2b5fe20c064d45c5cba17551ed2a4ffabd6bf493a354f: Status 404 returned error can't find the container with id 3a8ae512781e6df141b2b5fe20c064d45c5cba17551ed2a4ffabd6bf493a354f
Oct 30 22:21:43.087966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:43.087922    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr" event={"ID":"d802ee16-0512-4053-a1e2-d12ab06210f9","Type":"ContainerStarted","Data":"3a8ae512781e6df141b2b5fe20c064d45c5cba17551ed2a4ffabd6bf493a354f"}
Oct 30 22:21:44.102033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:44.101997    5031 generic.go:356] "Generic (PLEG): container finished" podID="d802ee16-0512-4053-a1e2-d12ab06210f9" containerID="3de486e61947e90320217495beb78942ed7ca68c17dccac688edbcc1606cd920" exitCode=0
Oct 30 22:21:44.102544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:44.102085    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr" event={"ID":"d802ee16-0512-4053-a1e2-d12ab06210f9","Type":"ContainerDied","Data":"3de486e61947e90320217495beb78942ed7ca68c17dccac688edbcc1606cd920"}
Oct 30 22:21:45.327493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.327465    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:45.434853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.434761    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-xks-ca\") pod \"d802ee16-0512-4053-a1e2-d12ab06210f9\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") "
Oct 30 22:21:45.435021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.434897    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d802ee16-0512-4053-a1e2-d12ab06210f9-kube-api-access-gsnnr\" (UniqueName: \"kubernetes.io/projected/d802ee16-0512-4053-a1e2-d12ab06210f9-kube-api-access-gsnnr\") pod \"d802ee16-0512-4053-a1e2-d12ab06210f9\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") "
Oct 30 22:21:45.435166 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.435149    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-openshift-ca\") pod \"d802ee16-0512-4053-a1e2-d12ab06210f9\" (UID: \"d802ee16-0512-4053-a1e2-d12ab06210f9\") "
Oct 30 22:21:45.435661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.435569    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-openshift-ca" pod "d802ee16-0512-4053-a1e2-d12ab06210f9" (UID: "d802ee16-0512-4053-a1e2-d12ab06210f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:45.436093 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.436070    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:45.437265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.437237    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-xks-ca" pod "d802ee16-0512-4053-a1e2-d12ab06210f9" (UID: "d802ee16-0512-4053-a1e2-d12ab06210f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:45.437455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.437432    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d802ee16-0512-4053-a1e2-d12ab06210f9-kube-api-access-gsnnr" pod "d802ee16-0512-4053-a1e2-d12ab06210f9" (UID: "d802ee16-0512-4053-a1e2-d12ab06210f9"). InnerVolumeSpecName "kube-api-access-gsnnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:45.537403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.537363    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d802ee16-0512-4053-a1e2-d12ab06210f9-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:45.537403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:45.537403    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnnr\" (UniqueName: \"kubernetes.io/projected/d802ee16-0512-4053-a1e2-d12ab06210f9-kube-api-access-gsnnr\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:46.124165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:46.124128    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"
Oct 30 22:21:46.124406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:46.124162    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr" event={"ID":"d802ee16-0512-4053-a1e2-d12ab06210f9","Type":"ContainerDied","Data":"3a8ae512781e6df141b2b5fe20c064d45c5cba17551ed2a4ffabd6bf493a354f"}
Oct 30 22:21:46.124406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:46.124261    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8ae512781e6df141b2b5fe20c064d45c5cba17551ed2a4ffabd6bf493a354f"
Oct 30 22:21:47.416428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:47.416366    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:49.830639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:49.830594    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:50.546374 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:21:50.546330    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:21:52.727248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:52.725331    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:52.922238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:52.922079    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:21:53.593083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:53.593039    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"]
Oct 30 22:21:53.594262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:53.594204    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy" containerID="cri-o://ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e" gracePeriod=30
Oct 30 22:21:53.645389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:53.645322    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:53.914329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:53.914199    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:57.885487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:57.885388    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:58.905027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:58.904957    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:21:59.015018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.014982    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-socket\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015044    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-envoy\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015159    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-podinfo\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015292    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-token\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-token\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015328    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1541b324-ba5a-41d5-b979-9ba0d860117a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541b324-ba5a-41d5-b979-9ba0d860117a-istiod-ca-cert\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015356    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-data\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015377    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-socket" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.015428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015407    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-credential-socket\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015525    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-kube-api-access-4nmkz\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-kube-api-access-4nmkz\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015581    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-certs\") pod \"1541b324-ba5a-41d5-b979-9ba0d860117a\" (UID: \"1541b324-ba5a-41d5-b979-9ba0d860117a\") "
Oct 30 22:21:59.015826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015790    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-credential-socket" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.016143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.016076    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-certs" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.016143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.015992    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1541b324-ba5a-41d5-b979-9ba0d860117a-istiod-ca-cert" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:59.016533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.016490    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1541b324-ba5a-41d5-b979-9ba0d860117a-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.016774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.016520    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.016774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.016564    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.016774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.016579    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.016774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.016584    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-data" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.018362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.018332    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-envoy" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:59.018474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.018439    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-token" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:59.018520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.018465    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-kube-api-access-4nmkz" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "kube-api-access-4nmkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:59.019625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.019597    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-podinfo" pod "1541b324-ba5a-41d5-b979-9ba0d860117a" (UID: "1541b324-ba5a-41d5-b979-9ba0d860117a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:59.118496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.118458    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmkz\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-kube-api-access-4nmkz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.118667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.118500    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.118667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.118530    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.118667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.118544    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.118667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.118558    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1541b324-ba5a-41d5-b979-9ba0d860117a-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:59.248439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.248354    5031 generic.go:356] "Generic (PLEG): container finished" podID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerID="ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e" exitCode=0
Oct 30 22:21:59.248439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.248406    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"
Oct 30 22:21:59.248750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.248431    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" event={"ID":"1541b324-ba5a-41d5-b979-9ba0d860117a","Type":"ContainerDied","Data":"ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e"}
Oct 30 22:21:59.248750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.248488    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs" event={"ID":"1541b324-ba5a-41d5-b979-9ba0d860117a","Type":"ContainerDied","Data":"b16692f9c606a79622413301162b40e4674a93a7a71003acb64ed49c7406ee09"}
Oct 30 22:21:59.248750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.248511    5031 scope.go:122] "RemoveContainer" containerID="ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e"
Oct 30 22:21:59.262372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.262353    5031 scope.go:122] "RemoveContainer" containerID="ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e"
Oct 30 22:21:59.262647 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:21:59.262623    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e\": container with ID starting with ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e not found: ID does not exist" containerID="ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e"
Oct 30 22:21:59.262748 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.262652    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e"} err="failed to get container status \"ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e\": rpc error: code = NotFound desc = could not find container \"ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e\": container with ID starting with ac4240db8e3e6e217ab82d663443ba246104f6abaefe8136fc785452cfd03e6e not found: ID does not exist"
Oct 30 22:21:59.425413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.425378    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"]
Oct 30 22:21:59.429922 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:21:59.429893    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs"]
Oct 30 22:22:00.650583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:00.650543    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" path="/var/lib/kubelet/pods/1541b324-ba5a-41d5-b979-9ba0d860117a/volumes"
Oct 30 22:22:02.839949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:02.839898    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:03.066302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:03.066244    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:03.335933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:03.335896    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.445313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:08.445280    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:13.247532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:13.247500    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:13.512433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:13.512342    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:13.762369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:13.762332    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:18.627104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:18.621109    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:20.543174 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:22:20.543140    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:22:23.907631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:23.907595    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:24.260567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:24.260477    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:24.510908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:24.510812    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:24.757338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:24.757289    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:28.892540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:28.892482    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:33.791173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:33.791131    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:34.067286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:34.067185    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:36.970852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:36.970805    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:22:38.311486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:38.311421    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"]
Oct 30 22:22:38.312339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:38.312302    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" containerID="cri-o://40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607" gracePeriod=30
Oct 30 22:22:38.839048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:38.839006    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:22:38.839282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:38.839085    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:39.254764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:39.254724    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:42.743859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:42.743820    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"]
Oct 30 22:22:42.744397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:42.744036    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy" containerID="cri-o://2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35" gracePeriod=30
Oct 30 22:22:43.619558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.619529    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:22:43.649955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.649921    5031 generic.go:356] "Generic (PLEG): container finished" podID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerID="40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607" exitCode=0
Oct 30 22:22:43.650145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.649976    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"
Oct 30 22:22:43.650145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.650006    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" event={"ID":"0dd502b3-3610-41ce-b58a-9e028f86e765","Type":"ContainerDied","Data":"40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607"}
Oct 30 22:22:43.650145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.650072    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2" event={"ID":"0dd502b3-3610-41ce-b58a-9e028f86e765","Type":"ContainerDied","Data":"4f833482007d8eeb761d12b6ad4ce4de74c5ec46b91434c5ee95e0da9d5f68d9"}
Oct 30 22:22:43.650145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.650096    5031 scope.go:122] "RemoveContainer" containerID="40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607"
Oct 30 22:22:43.662273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.662254    5031 scope.go:122] "RemoveContainer" containerID="40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607"
Oct 30 22:22:43.662588 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:22:43.662569    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607\": container with ID starting with 40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607 not found: ID does not exist" containerID="40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607"
Oct 30 22:22:43.662644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.662597    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607"} err="failed to get container status \"40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607\": rpc error: code = NotFound desc = could not find container \"40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607\": container with ID starting with 40270ecf7fd27a3874edc3e52b61b14ba182faf0990b8a1e4333becfedcc0607 not found: ID does not exist"
Oct 30 22:22:43.784808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.784767    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-data\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.785472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.785024    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-data" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:43.785655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.785488    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-podinfo\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.785772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.785711    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-token\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-token\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.785841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.785769    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-certs\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.785915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.785895    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-credential-socket\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.786033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.786015    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-socket\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.787277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.786359    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-socket" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:43.787277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.786659    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-certs" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:43.787277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.786814    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd502b3-3610-41ce-b58a-9e028f86e765-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd502b3-3610-41ce-b58a-9e028f86e765-istiod-ca-cert\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.787277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.786874    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-kube-api-access-2s2bt\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-kube-api-access-2s2bt\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.787277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.787049    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-envoy\") pod \"0dd502b3-3610-41ce-b58a-9e028f86e765\" (UID: \"0dd502b3-3610-41ce-b58a-9e028f86e765\") "
Oct 30 22:22:43.787277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.787192    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd502b3-3610-41ce-b58a-9e028f86e765-istiod-ca-cert" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:43.787746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.787393    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-credential-socket" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:43.788570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.788538    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.788666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.788575    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd502b3-3610-41ce-b58a-9e028f86e765-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.788666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.788596    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.788666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.788613    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.788666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.788629    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.789929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.789888    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-podinfo" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:43.790047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.789933    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-token" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:43.790831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.790790    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-kube-api-access-2s2bt" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "kube-api-access-2s2bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:43.791362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.791335    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-envoy" pod "0dd502b3-3610-41ce-b58a-9e028f86e765" (UID: "0dd502b3-3610-41ce-b58a-9e028f86e765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:43.889873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.889834    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.889873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.889867    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.889873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.889877    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:43.890174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:43.889887    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2bt\" (UniqueName: \"kubernetes.io/projected/0dd502b3-3610-41ce-b58a-9e028f86e765-kube-api-access-2s2bt\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:44.079357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:44.079317    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"]
Oct 30 22:22:44.087673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:44.087635    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2"]
Oct 30 22:22:44.409122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:44.409082    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:44.649541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:44.649496    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" path="/var/lib/kubelet/pods/0dd502b3-3610-41ce-b58a-9e028f86e765/volumes"
Oct 30 22:22:48.088781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.088751    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:22:48.156388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156352    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-credential-socket\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.156598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156435    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-socket\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.156598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156486    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-kube-api-access-tvv26\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-kube-api-access-tvv26\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.156598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156514    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-envoy\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.156598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156573    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-data\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.156847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156701    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-credential-socket" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:48.156847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156726    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-certs\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.156847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156763    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-socket" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:48.156847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156822    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6530c813-1262-4765-8920-e190736f90b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6530c813-1262-4765-8920-e190736f90b2-istio-podinfo\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.157069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156902    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-data" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:48.157069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156914    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-certs" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:48.157069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.156930    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-istio-token\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-istio-token\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.157069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.157008    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6530c813-1262-4765-8920-e190736f90b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6530c813-1262-4765-8920-e190736f90b2-istiod-ca-cert\") pod \"6530c813-1262-4765-8920-e190736f90b2\" (UID: \"6530c813-1262-4765-8920-e190736f90b2\") "
Oct 30 22:22:48.157484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.157452    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6530c813-1262-4765-8920-e190736f90b2-istiod-ca-cert" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:48.157773 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.157753    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6530c813-1262-4765-8920-e190736f90b2-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.157867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.157779    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.157867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.157793    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.157867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.157807    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.157867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.157820    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.159327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.159285    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6530c813-1262-4765-8920-e190736f90b2-istio-podinfo" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:48.159425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.159405    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-envoy" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:48.159762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.159736    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-istio-token" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:48.159866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.159844    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-kube-api-access-tvv26" pod "6530c813-1262-4765-8920-e190736f90b2" (UID: "6530c813-1262-4765-8920-e190736f90b2"). InnerVolumeSpecName "kube-api-access-tvv26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:48.259640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.259603    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvv26\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-kube-api-access-tvv26\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.259640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.259642    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6530c813-1262-4765-8920-e190736f90b2-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.259866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.259658    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6530c813-1262-4765-8920-e190736f90b2-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.259866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.259672    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6530c813-1262-4765-8920-e190736f90b2-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:48.698945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.698912    5031 generic.go:356] "Generic (PLEG): container finished" podID="6530c813-1262-4765-8920-e190736f90b2" containerID="2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35" exitCode=0
Oct 30 22:22:48.699124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.698962    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"
Oct 30 22:22:48.699124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.698988    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" event={"ID":"6530c813-1262-4765-8920-e190736f90b2","Type":"ContainerDied","Data":"2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35"}
Oct 30 22:22:48.699124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.699043    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc" event={"ID":"6530c813-1262-4765-8920-e190736f90b2","Type":"ContainerDied","Data":"250601ffc31b136d9775fa4257d28d50ab0b762d10e46dcf8cc527836cfbab2a"}
Oct 30 22:22:48.699124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.699068    5031 scope.go:122] "RemoveContainer" containerID="2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35"
Oct 30 22:22:48.718562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.718543    5031 scope.go:122] "RemoveContainer" containerID="2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35"
Oct 30 22:22:48.718894 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:22:48.718874    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35\": container with ID starting with 2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35 not found: ID does not exist" containerID="2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35"
Oct 30 22:22:48.718959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.718905    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35"} err="failed to get container status \"2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35\": rpc error: code = NotFound desc = could not find container \"2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35\": container with ID starting with 2d5818add84b72bdbe9392f805696a433dd5d6e4298392bee8bc91d4c87a9c35 not found: ID does not exist"
Oct 30 22:22:48.875167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.875123    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"]
Oct 30 22:22:48.928567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:48.928484    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc"]
Oct 30 22:22:49.521572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:49.521529    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:50.272314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:50.272276    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:50.543007 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:22:50.542921    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:22:50.543475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:50.543156    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:50.648351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:50.648315    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6530c813-1262-4765-8920-e190736f90b2" path="/var/lib/kubelet/pods/6530c813-1262-4765-8920-e190736f90b2/volumes"
Oct 30 22:22:51.156252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.156218    5031 scope.go:122] "RemoveContainer" containerID="3b59226aff82511b3be49342853da75853fd26a911f7f17eaa323bb929db46fb"
Oct 30 22:22:51.168259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.168220    5031 scope.go:122] "RemoveContainer" containerID="d0b4ecfbea22c0f8e6e4df4ae10f8d74ce44e3401ff7ba2bb099fc6514bb1298"
Oct 30 22:22:51.306766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.306723    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"]
Oct 30 22:22:51.307947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.307918    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy"
Oct 30 22:22:51.307947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.307947    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy"
Oct 30 22:22:51.308106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.307975    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy"
Oct 30 22:22:51.308106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.307983    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy"
Oct 30 22:22:51.308106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308010    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d802ee16-0512-4053-a1e2-d12ab06210f9" containerName="revoke-keys"
Oct 30 22:22:51.308106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308017    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d802ee16-0512-4053-a1e2-d12ab06210f9" containerName="revoke-keys"
Oct 30 22:22:51.308106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308043    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy"
Oct 30 22:22:51.308106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308049    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy"
Oct 30 22:22:51.308334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308246    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1541b324-ba5a-41d5-b979-9ba0d860117a" containerName="istio-proxy"
Oct 30 22:22:51.308334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308265    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd502b3-3610-41ce-b58a-9e028f86e765" containerName="istio-proxy"
Oct 30 22:22:51.308334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308279    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d802ee16-0512-4053-a1e2-d12ab06210f9" containerName="revoke-keys"
Oct 30 22:22:51.308334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.308292    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6530c813-1262-4765-8920-e190736f90b2" containerName="istio-proxy"
Oct 30 22:22:51.314524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.314500    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.369259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.369188    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"]
Oct 30 22:22:51.410304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.410166    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/b1821963-2caf-4b69-818b-a4cff2901fd9-kube-api-access-4w5rv\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.410494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.410400    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.410669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.410646    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.414221 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.414190    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:51.414376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.414268    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:51.414376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.414354    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:51.419812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.419786    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:22:51.511988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.511950    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.512172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.512031    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/b1821963-2caf-4b69-818b-a4cff2901fd9-kube-api-access-4w5rv\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.512172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.512136    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.512714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.512695    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.514764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.514745    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.588999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.588958    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/b1821963-2caf-4b69-818b-a4cff2901fd9-kube-api-access-4w5rv\") pod \"maas-api-revoke-keys-e2e-api-a-81f37a-vspsq\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:51.629964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:51.629924    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:52.021978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:52.021945    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"]
Oct 30 22:22:52.025958 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:22:52.025911    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1821963_2caf_4b69_818b_a4cff2901fd9.slice/crio-e480c96b7e24ef89a59a99673f99212fcbda763b6c90d3f4a8e293ab65cc09bc WatchSource:0}: Error finding container e480c96b7e24ef89a59a99673f99212fcbda763b6c90d3f4a8e293ab65cc09bc: Status 404 returned error can't find the container with id e480c96b7e24ef89a59a99673f99212fcbda763b6c90d3f4a8e293ab65cc09bc
Oct 30 22:22:52.749331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:52.745313    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq" event={"ID":"b1821963-2caf-4b69-818b-a4cff2901fd9","Type":"ContainerStarted","Data":"1b9504e2333b61aadf7d6145b0d0b06f390073d11b920aa0e0c7bcf4574144f2"}
Oct 30 22:22:52.749331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:52.745376    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq" event={"ID":"b1821963-2caf-4b69-818b-a4cff2901fd9","Type":"ContainerStarted","Data":"e480c96b7e24ef89a59a99673f99212fcbda763b6c90d3f4a8e293ab65cc09bc"}
Oct 30 22:22:52.867594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:52.867539    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq" podStartSLOduration=1.867523865 podStartE2EDuration="1.867523865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:52.866781203 +0000 UTC m=+1984.390923986" watchObservedRunningTime="2026-10-30 22:22:52.867523865 +0000 UTC m=+1984.391666629"
Oct 30 22:22:53.759089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:53.759053    5031 generic.go:356] "Generic (PLEG): container finished" podID="b1821963-2caf-4b69-818b-a4cff2901fd9" containerID="1b9504e2333b61aadf7d6145b0d0b06f390073d11b920aa0e0c7bcf4574144f2" exitCode=0
Oct 30 22:22:53.759557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:53.759140    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq" event={"ID":"b1821963-2caf-4b69-818b-a4cff2901fd9","Type":"ContainerDied","Data":"1b9504e2333b61aadf7d6145b0d0b06f390073d11b920aa0e0c7bcf4574144f2"}
Oct 30 22:22:54.975077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:54.975047    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:55.071287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.069042    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-openshift-ca\") pod \"b1821963-2caf-4b69-818b-a4cff2901fd9\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") "
Oct 30 22:22:55.071287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.069157    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1821963-2caf-4b69-818b-a4cff2901fd9-kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/b1821963-2caf-4b69-818b-a4cff2901fd9-kube-api-access-4w5rv\") pod \"b1821963-2caf-4b69-818b-a4cff2901fd9\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") "
Oct 30 22:22:55.071287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.069226    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-xks-ca\") pod \"b1821963-2caf-4b69-818b-a4cff2901fd9\" (UID: \"b1821963-2caf-4b69-818b-a4cff2901fd9\") "
Oct 30 22:22:55.071287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.069682    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-openshift-ca" pod "b1821963-2caf-4b69-818b-a4cff2901fd9" (UID: "b1821963-2caf-4b69-818b-a4cff2901fd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:55.077243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.074059    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1821963-2caf-4b69-818b-a4cff2901fd9-kube-api-access-4w5rv" pod "b1821963-2caf-4b69-818b-a4cff2901fd9" (UID: "b1821963-2caf-4b69-818b-a4cff2901fd9"). InnerVolumeSpecName "kube-api-access-4w5rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:55.077243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.075753    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:55.077243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.075783    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/b1821963-2caf-4b69-818b-a4cff2901fd9-kube-api-access-4w5rv\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:55.080732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.079486    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:55.080732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.079900    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-xks-ca" pod "b1821963-2caf-4b69-818b-a4cff2901fd9" (UID: "b1821963-2caf-4b69-818b-a4cff2901fd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:55.177202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.177161    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1821963-2caf-4b69-818b-a4cff2901fd9-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:55.298511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.298472    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"]
Oct 30 22:22:55.299296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.299275    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b1821963-2caf-4b69-818b-a4cff2901fd9" containerName="revoke-keys"
Oct 30 22:22:55.299296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.299293    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1821963-2caf-4b69-818b-a4cff2901fd9" containerName="revoke-keys"
Oct 30 22:22:55.299557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.299412    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1821963-2caf-4b69-818b-a4cff2901fd9" containerName="revoke-keys"
Oct 30 22:22:55.303404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.303385    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.365322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.365290    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"]
Oct 30 22:22:55.380140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.380105    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.380571 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.380546    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6jw\" (UniqueName: \"kubernetes.io/projected/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-kube-api-access-pk6jw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.380635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.380591    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.482352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.482321    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6jw\" (UniqueName: \"kubernetes.io/projected/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-kube-api-access-pk6jw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.482352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.482356    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.482702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.482399    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.483149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.483128    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.484990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.484970    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.546495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.546460    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6jw\" (UniqueName: \"kubernetes.io/projected/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-kube-api-access-pk6jw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.620938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.620858    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:55.777857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.777826    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"
Oct 30 22:22:55.777857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.777840    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq" event={"ID":"b1821963-2caf-4b69-818b-a4cff2901fd9","Type":"ContainerDied","Data":"e480c96b7e24ef89a59a99673f99212fcbda763b6c90d3f4a8e293ab65cc09bc"}
Oct 30 22:22:55.778073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:55.777891    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e480c96b7e24ef89a59a99673f99212fcbda763b6c90d3f4a8e293ab65cc09bc"
Oct 30 22:22:56.039604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:56.039573    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"]
Oct 30 22:22:56.040799 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:22:56.040763    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602bf24f_1360_4ff4_b0d3_6ba1e791ecdb.slice/crio-9bab2a8bf9757dba17e361bd19a54bf3d6811ab870b46db8b5ca9f26a3832052 WatchSource:0}: Error finding container 9bab2a8bf9757dba17e361bd19a54bf3d6811ab870b46db8b5ca9f26a3832052: Status 404 returned error can't find the container with id 9bab2a8bf9757dba17e361bd19a54bf3d6811ab870b46db8b5ca9f26a3832052
Oct 30 22:22:56.787606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:56.787496    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b" event={"ID":"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb","Type":"ContainerStarted","Data":"9acd14060e360d49402bc4b507203a6b479c7678f80e19f8149f1e58f23259ac"}
Oct 30 22:22:56.787606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:56.787574    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b" event={"ID":"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb","Type":"ContainerStarted","Data":"9bab2a8bf9757dba17e361bd19a54bf3d6811ab870b46db8b5ca9f26a3832052"}
Oct 30 22:22:56.909475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:56.909409    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b" podStartSLOduration=1.909391235 podStartE2EDuration="1.909391235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:56.907896753 +0000 UTC m=+1988.432039519" watchObservedRunningTime="2026-10-30 22:22:56.909391235 +0000 UTC m=+1988.433534000"
Oct 30 22:22:57.798132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:57.798093    5031 generic.go:356] "Generic (PLEG): container finished" podID="602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" containerID="9acd14060e360d49402bc4b507203a6b479c7678f80e19f8149f1e58f23259ac" exitCode=0
Oct 30 22:22:57.798639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:57.798179    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b" event={"ID":"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb","Type":"ContainerDied","Data":"9acd14060e360d49402bc4b507203a6b479c7678f80e19f8149f1e58f23259ac"}
Oct 30 22:22:59.020202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.020171    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:59.137741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.137702    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-openshift-ca\") pod \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") "
Oct 30 22:22:59.137937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.137869    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-xks-ca\") pod \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") "
Oct 30 22:22:59.138096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.138077    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-kube-api-access-pk6jw\" (UniqueName: \"kubernetes.io/projected/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-kube-api-access-pk6jw\") pod \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\" (UID: \"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb\") "
Oct 30 22:22:59.138165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.138137    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-openshift-ca" pod "602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" (UID: "602bf24f-1360-4ff4-b0d3-6ba1e791ecdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:59.138677 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.138650    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:59.140719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.140692    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-kube-api-access-pk6jw" pod "602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" (UID: "602bf24f-1360-4ff4-b0d3-6ba1e791ecdb"). InnerVolumeSpecName "kube-api-access-pk6jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:59.143042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.142494    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-xks-ca" pod "602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" (UID: "602bf24f-1360-4ff4-b0d3-6ba1e791ecdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:59.239798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.239762    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk6jw\" (UniqueName: \"kubernetes.io/projected/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-kube-api-access-pk6jw\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:59.239798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.239795    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:59.819912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.819869    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"
Oct 30 22:22:59.820095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.819872    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b" event={"ID":"602bf24f-1360-4ff4-b0d3-6ba1e791ecdb","Type":"ContainerDied","Data":"9bab2a8bf9757dba17e361bd19a54bf3d6811ab870b46db8b5ca9f26a3832052"}
Oct 30 22:22:59.820095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:22:59.820007    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bab2a8bf9757dba17e361bd19a54bf3d6811ab870b46db8b5ca9f26a3832052"
Oct 30 22:23:00.372728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:00.372690    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.140237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:02.139999    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.531583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:02.531471    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:05.642706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:05.642656    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:05.889035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:05.888886    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.119875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.119842    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.431101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.431009    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"]
Oct 30 22:23:06.431859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.431841    5031 cpu_manager.go:414] "containerMap: removing container" podUID="602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" containerName="revoke-keys"
Oct 30 22:23:06.431912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.431860    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" containerName="revoke-keys"
Oct 30 22:23:06.432000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.431991    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" containerName="revoke-keys"
Oct 30 22:23:06.438480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.438444    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.494194 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.494161    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"]
Oct 30 22:23:06.510534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.510490    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.510754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.510698    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.510953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.510918    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/d3c4acc6-714a-464c-bea6-6b9733903a62-kube-api-access-tbmfq\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.539021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.538986    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:06.539021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.538993    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:23:06.539021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.539004    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:06.543370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.543349    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:06.612741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.612703    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.613199 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.613169    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/d3c4acc6-714a-464c-bea6-6b9733903a62-kube-api-access-tbmfq\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.613355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.613329    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.613485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.613463    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.616046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.616017    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.676811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.676774    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/d3c4acc6-714a-464c-bea6-6b9733903a62-kube-api-access-tbmfq\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:06.761118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:06.761027    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:07.161604 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:23:07.161567    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c4acc6_714a_464c_bea6_6b9733903a62.slice/crio-aebd7b61e5f3ecfbb9353e0fec5582702b7874da89c082a6f0f8cfe77808f43a WatchSource:0}: Error finding container aebd7b61e5f3ecfbb9353e0fec5582702b7874da89c082a6f0f8cfe77808f43a: Status 404 returned error can't find the container with id aebd7b61e5f3ecfbb9353e0fec5582702b7874da89c082a6f0f8cfe77808f43a
Oct 30 22:23:07.162283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:07.162257    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"]
Oct 30 22:23:07.337803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:07.337766    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.571148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:07.571062    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.940869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:07.940825    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw" event={"ID":"d3c4acc6-714a-464c-bea6-6b9733903a62","Type":"ContainerStarted","Data":"53bda1535f10981630f3484fe82f9323eb9837957b675ba45b6566118fe239a9"}
Oct 30 22:23:07.941722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:07.941690    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw" event={"ID":"d3c4acc6-714a-464c-bea6-6b9733903a62","Type":"ContainerStarted","Data":"aebd7b61e5f3ecfbb9353e0fec5582702b7874da89c082a6f0f8cfe77808f43a"}
Oct 30 22:23:08.076484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:08.076398    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw" podStartSLOduration=2.076377578 podStartE2EDuration="2.076377578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:08.067743429 +0000 UTC m=+1999.591886210" watchObservedRunningTime="2026-10-30 22:23:08.076377578 +0000 UTC m=+1999.600520342"
Oct 30 22:23:08.953424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:08.953384    5031 generic.go:356] "Generic (PLEG): container finished" podID="d3c4acc6-714a-464c-bea6-6b9733903a62" containerID="53bda1535f10981630f3484fe82f9323eb9837957b675ba45b6566118fe239a9" exitCode=0
Oct 30 22:23:08.953970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:08.953497    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw" event={"ID":"d3c4acc6-714a-464c-bea6-6b9733903a62","Type":"ContainerDied","Data":"53bda1535f10981630f3484fe82f9323eb9837957b675ba45b6566118fe239a9"}
Oct 30 22:23:09.542435 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:09.542386    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"]
Oct 30 22:23:09.650703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:09.650652    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"]
Oct 30 22:23:10.031233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.024558    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.336254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.336118    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.344717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.344690    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:10.444318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.444272    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-xks-ca\") pod \"d3c4acc6-714a-464c-bea6-6b9733903a62\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") "
Oct 30 22:23:10.444525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.444389    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-openshift-ca\") pod \"d3c4acc6-714a-464c-bea6-6b9733903a62\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") "
Oct 30 22:23:10.444590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.444540    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c4acc6-714a-464c-bea6-6b9733903a62-kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/d3c4acc6-714a-464c-bea6-6b9733903a62-kube-api-access-tbmfq\") pod \"d3c4acc6-714a-464c-bea6-6b9733903a62\" (UID: \"d3c4acc6-714a-464c-bea6-6b9733903a62\") "
Oct 30 22:23:10.445064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.445037    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-openshift-ca" pod "d3c4acc6-714a-464c-bea6-6b9733903a62" (UID: "d3c4acc6-714a-464c-bea6-6b9733903a62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:10.452267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.447347    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-xks-ca" pod "d3c4acc6-714a-464c-bea6-6b9733903a62" (UID: "d3c4acc6-714a-464c-bea6-6b9733903a62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:10.452267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.447426    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c4acc6-714a-464c-bea6-6b9733903a62-kube-api-access-tbmfq" pod "d3c4acc6-714a-464c-bea6-6b9733903a62" (UID: "d3c4acc6-714a-464c-bea6-6b9733903a62"). InnerVolumeSpecName "kube-api-access-tbmfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:10.546640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.546544    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/d3c4acc6-714a-464c-bea6-6b9733903a62-kube-api-access-tbmfq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.546640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.546579    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.546640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.546590    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c4acc6-714a-464c-bea6-6b9733903a62-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.974790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.974758    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"
Oct 30 22:23:10.975021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.974786    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw" event={"ID":"d3c4acc6-714a-464c-bea6-6b9733903a62","Type":"ContainerDied","Data":"aebd7b61e5f3ecfbb9353e0fec5582702b7874da89c082a6f0f8cfe77808f43a"}
Oct 30 22:23:10.975021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:10.974841    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aebd7b61e5f3ecfbb9353e0fec5582702b7874da89c082a6f0f8cfe77808f43a"
Oct 30 22:23:11.640230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:11.639486    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:14.543479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:14.543397    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb" podUID="98118413-b24e-4f80-8fe9-15a09b89ef98" containerName="payload-processing" containerID="cri-o://26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0" gracePeriod=25
Oct 30 22:23:14.651958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:14.651916    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96" podUID="e4428c1b-22ee-4577-8786-a9c399f57154" containerName="payload-pre-processing" containerID="cri-o://8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd" gracePeriod=25
Oct 30 22:23:14.908692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:14.908655    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:23:14.982591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:14.982543    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:23:15.024787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.024746    5031 generic.go:356] "Generic (PLEG): container finished" podID="e4428c1b-22ee-4577-8786-a9c399f57154" containerID="8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd" exitCode=0
Oct 30 22:23:15.024985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.024799    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"
Oct 30 22:23:15.024985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.024835    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96" event={"ID":"e4428c1b-22ee-4577-8786-a9c399f57154","Type":"ContainerDied","Data":"8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd"}
Oct 30 22:23:15.024985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.024907    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96" event={"ID":"e4428c1b-22ee-4577-8786-a9c399f57154","Type":"ContainerDied","Data":"f04cb7d1b5420fb9efdb4a3f2cc439a9b06e52faa532f7144788ddaa13316779"}
Oct 30 22:23:15.024985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.024938    5031 scope.go:122] "RemoveContainer" containerID="8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd"
Oct 30 22:23:15.026634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.026605    5031 generic.go:356] "Generic (PLEG): container finished" podID="98118413-b24e-4f80-8fe9-15a09b89ef98" containerID="26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0" exitCode=0
Oct 30 22:23:15.026780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.026721    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb" event={"ID":"98118413-b24e-4f80-8fe9-15a09b89ef98","Type":"ContainerDied","Data":"26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0"}
Oct 30 22:23:15.026780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.026773    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb" event={"ID":"98118413-b24e-4f80-8fe9-15a09b89ef98","Type":"ContainerDied","Data":"5ccd7ec94f0847cd5954f19e824d8948ac651b3463a6d72894113a5d8fd75fb8"}
Oct 30 22:23:15.026872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.026808    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"
Oct 30 22:23:15.029084 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.029061    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98118413-b24e-4f80-8fe9-15a09b89ef98-kube-api-access-2tprm\" (UniqueName: \"kubernetes.io/projected/98118413-b24e-4f80-8fe9-15a09b89ef98-kube-api-access-2tprm\") pod \"98118413-b24e-4f80-8fe9-15a09b89ef98\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") "
Oct 30 22:23:15.029204 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.029187    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4428c1b-22ee-4577-8786-a9c399f57154-kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/e4428c1b-22ee-4577-8786-a9c399f57154-kube-api-access-6rjcc\") pod \"e4428c1b-22ee-4577-8786-a9c399f57154\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") "
Oct 30 22:23:15.029295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.029240    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98118413-b24e-4f80-8fe9-15a09b89ef98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98118413-b24e-4f80-8fe9-15a09b89ef98-plugins-config-volume\") pod \"98118413-b24e-4f80-8fe9-15a09b89ef98\" (UID: \"98118413-b24e-4f80-8fe9-15a09b89ef98\") "
Oct 30 22:23:15.029385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.029368    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4428c1b-22ee-4577-8786-a9c399f57154-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4428c1b-22ee-4577-8786-a9c399f57154-plugins-config-volume\") pod \"e4428c1b-22ee-4577-8786-a9c399f57154\" (UID: \"e4428c1b-22ee-4577-8786-a9c399f57154\") "
Oct 30 22:23:15.035580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.035540    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98118413-b24e-4f80-8fe9-15a09b89ef98-kube-api-access-2tprm" pod "98118413-b24e-4f80-8fe9-15a09b89ef98" (UID: "98118413-b24e-4f80-8fe9-15a09b89ef98"). InnerVolumeSpecName "kube-api-access-2tprm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:15.036295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.036265    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4428c1b-22ee-4577-8786-a9c399f57154-plugins-config-volume" pod "e4428c1b-22ee-4577-8786-a9c399f57154" (UID: "e4428c1b-22ee-4577-8786-a9c399f57154"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:15.037540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.037516    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4428c1b-22ee-4577-8786-a9c399f57154-kube-api-access-6rjcc" pod "e4428c1b-22ee-4577-8786-a9c399f57154" (UID: "e4428c1b-22ee-4577-8786-a9c399f57154"). InnerVolumeSpecName "kube-api-access-6rjcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:15.037630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.037562    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98118413-b24e-4f80-8fe9-15a09b89ef98-plugins-config-volume" pod "98118413-b24e-4f80-8fe9-15a09b89ef98" (UID: "98118413-b24e-4f80-8fe9-15a09b89ef98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:15.041054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.041032    5031 scope.go:122] "RemoveContainer" containerID="8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd"
Oct 30 22:23:15.041458 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:23:15.041428    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd\": container with ID starting with 8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd not found: ID does not exist" containerID="8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd"
Oct 30 22:23:15.041539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.041469    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd"} err="failed to get container status \"8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd\": rpc error: code = NotFound desc = could not find container \"8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd\": container with ID starting with 8ca6fc47ea7e39a8b1ab1411d6041710839d35e2271f2afcddc12a458cd819fd not found: ID does not exist"
Oct 30 22:23:15.041539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.041495    5031 scope.go:122] "RemoveContainer" containerID="26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0"
Oct 30 22:23:15.055016 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.054993    5031 scope.go:122] "RemoveContainer" containerID="26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0"
Oct 30 22:23:15.055464 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:23:15.055441    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0\": container with ID starting with 26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0 not found: ID does not exist" containerID="26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0"
Oct 30 22:23:15.055616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.055477    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0"} err="failed to get container status \"26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0\": rpc error: code = NotFound desc = could not find container \"26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0\": container with ID starting with 26a6e0165283e677638955d4c330387bbe87a32700f0cf7a0c641990c371aeb0 not found: ID does not exist"
Oct 30 22:23:15.132995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.132892    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tprm\" (UniqueName: \"kubernetes.io/projected/98118413-b24e-4f80-8fe9-15a09b89ef98-kube-api-access-2tprm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.132995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.132940    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/e4428c1b-22ee-4577-8786-a9c399f57154-kube-api-access-6rjcc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.132995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.132954    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98118413-b24e-4f80-8fe9-15a09b89ef98-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.132995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.132970    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4428c1b-22ee-4577-8786-a9c399f57154-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.457302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.457204    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"]
Oct 30 22:23:15.465845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.465804    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96"]
Oct 30 22:23:15.579245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.579193    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"]
Oct 30 22:23:15.584606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:15.584574    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb"]
Oct 30 22:23:16.650756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:16.650722    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98118413-b24e-4f80-8fe9-15a09b89ef98" path="/var/lib/kubelet/pods/98118413-b24e-4f80-8fe9-15a09b89ef98/volumes"
Oct 30 22:23:16.651356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:16.651189    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4428c1b-22ee-4577-8786-a9c399f57154" path="/var/lib/kubelet/pods/e4428c1b-22ee-4577-8786-a9c399f57154/volumes"
Oct 30 22:23:16.822115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:16.822021    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"]
Oct 30 22:23:16.964235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:16.960296    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"]
Oct 30 22:23:16.964235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:16.963544    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:17.055299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.055197    5031 generic.go:356] "Generic (PLEG): container finished" podID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerID="b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd" exitCode=0
Oct 30 22:23:17.055512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.055374    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" event={"ID":"f6fcb1e3-b93e-42e8-b489-bbec1d3409de","Type":"ContainerDied","Data":"b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd"}
Oct 30 22:23:17.055885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.055868    5031 scope.go:122] "RemoveContainer" containerID="b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd"
Oct 30 22:23:17.057851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.057819    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:17.057957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.057908    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:17.058037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.057978    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:17.310808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.309650    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:17.539930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:17.539897    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:18.066895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:18.066860    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" event={"ID":"f6fcb1e3-b93e-42e8-b489-bbec1d3409de","Type":"ContainerStarted","Data":"f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3"}
Oct 30 22:23:18.558634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:18.558602    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:23:18.559078 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:18.558981    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:23:18.559259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:18.559233    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:23:19.556316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:19.556279    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:23:19.556756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:19.556353    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:23:20.555985 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:20.555948    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:23:20.556235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:20.556019    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:23:20.567311 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:23:20.567273    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:23:21.395906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.395866    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"]
Oct 30 22:23:21.397401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.397368    5031 cpu_manager.go:414] "containerMap: removing container" podUID="98118413-b24e-4f80-8fe9-15a09b89ef98" containerName="payload-processing"
Oct 30 22:23:21.397604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.397585    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98118413-b24e-4f80-8fe9-15a09b89ef98" containerName="payload-processing"
Oct 30 22:23:21.397735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.397720    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d3c4acc6-714a-464c-bea6-6b9733903a62" containerName="revoke-keys"
Oct 30 22:23:21.397843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.397829    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c4acc6-714a-464c-bea6-6b9733903a62" containerName="revoke-keys"
Oct 30 22:23:21.397956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.397943    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e4428c1b-22ee-4577-8786-a9c399f57154" containerName="payload-pre-processing"
Oct 30 22:23:21.398048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.398034    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4428c1b-22ee-4577-8786-a9c399f57154" containerName="payload-pre-processing"
Oct 30 22:23:21.403576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.403548    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="98118413-b24e-4f80-8fe9-15a09b89ef98" containerName="payload-processing"
Oct 30 22:23:21.403747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.403734    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c4acc6-714a-464c-bea6-6b9733903a62" containerName="revoke-keys"
Oct 30 22:23:21.403857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.403842    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4428c1b-22ee-4577-8786-a9c399f57154" containerName="payload-pre-processing"
Oct 30 22:23:21.421667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.421640    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.492256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.492191    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.497493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.497462    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bst9p\" (UniqueName: \"kubernetes.io/projected/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-kube-api-access-bst9p\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.497683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.497559    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.497683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.496469    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:21.497683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.497179    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"]
Oct 30 22:23:21.497833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.496621    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:23:21.497833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.496631    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:21.537240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.530413    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:21.557281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.557249    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:23:21.557556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.557528    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:23:21.598790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.598744    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst9p\" (UniqueName: \"kubernetes.io/projected/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-kube-api-access-bst9p\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.599187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.598862    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.599187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.598998    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.599855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.599821    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.601730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.601697    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.664539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.664443    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bst9p\" (UniqueName: \"kubernetes.io/projected/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-kube-api-access-bst9p\") pod \"maas-api-revoke-sub-058f09b6-4fljx\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.781770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.781731    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:21.829250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.826276    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k" podUID="ba87d160-7551-4ea4-a296-ebf9155b5b5b" containerName="payload-processing" containerID="cri-o://fafce537e2c070101c8fdd205cbc95c0949a89ef44dff543acac85c9eb28ce28" gracePeriod=25
Oct 30 22:23:21.961727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:21.961499    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g" podUID="62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" containerName="payload-pre-processing" containerID="cri-o://1c538834c1fd158068dc991e97fe09fcfec45174ca27b5a5586449155a727955" gracePeriod=25
Oct 30 22:23:22.183993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.183951    5031 generic.go:356] "Generic (PLEG): container finished" podID="ba87d160-7551-4ea4-a296-ebf9155b5b5b" containerID="fafce537e2c070101c8fdd205cbc95c0949a89ef44dff543acac85c9eb28ce28" exitCode=0
Oct 30 22:23:22.184401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.184373    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k" event={"ID":"ba87d160-7551-4ea4-a296-ebf9155b5b5b","Type":"ContainerDied","Data":"fafce537e2c070101c8fdd205cbc95c0949a89ef44dff543acac85c9eb28ce28"}
Oct 30 22:23:22.264097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.264062    5031 generic.go:356] "Generic (PLEG): container finished" podID="62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" containerID="1c538834c1fd158068dc991e97fe09fcfec45174ca27b5a5586449155a727955" exitCode=0
Oct 30 22:23:22.264282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.264151    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g" event={"ID":"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e","Type":"ContainerDied","Data":"1c538834c1fd158068dc991e97fe09fcfec45174ca27b5a5586449155a727955"}
Oct 30 22:23:22.270095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.269958    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:22.273323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.270417    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"]
Oct 30 22:23:22.274591 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:23:22.274561    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e750cc_2e81_444a_b2be_fac2ca9f2e5f.slice/crio-d1f8bddbdf6ce8f3d11b601816bb3ec5bdb55571eebd3831719bf22dab4b3cc0 WatchSource:0}: Error finding container d1f8bddbdf6ce8f3d11b601816bb3ec5bdb55571eebd3831719bf22dab4b3cc0: Status 404 returned error can't find the container with id d1f8bddbdf6ce8f3d11b601816bb3ec5bdb55571eebd3831719bf22dab4b3cc0
Oct 30 22:23:22.277097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.277075    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:23:22.364798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.364765    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba87d160-7551-4ea4-a296-ebf9155b5b5b-kube-api-access-gf5lg\" (UniqueName: \"kubernetes.io/projected/ba87d160-7551-4ea4-a296-ebf9155b5b5b-kube-api-access-gf5lg\") pod \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") "
Oct 30 22:23:22.365000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.364987    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba87d160-7551-4ea4-a296-ebf9155b5b5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba87d160-7551-4ea4-a296-ebf9155b5b5b-plugins-config-volume\") pod \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\" (UID: \"ba87d160-7551-4ea4-a296-ebf9155b5b5b\") "
Oct 30 22:23:22.365932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.365904    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba87d160-7551-4ea4-a296-ebf9155b5b5b-plugins-config-volume" pod "ba87d160-7551-4ea4-a296-ebf9155b5b5b" (UID: "ba87d160-7551-4ea4-a296-ebf9155b5b5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:22.368834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.368801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba87d160-7551-4ea4-a296-ebf9155b5b5b-kube-api-access-gf5lg" pod "ba87d160-7551-4ea4-a296-ebf9155b5b5b" (UID: "ba87d160-7551-4ea4-a296-ebf9155b5b5b"). InnerVolumeSpecName "kube-api-access-gf5lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:22.460483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.460453    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:23:22.469072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.468631    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba87d160-7551-4ea4-a296-ebf9155b5b5b-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:22.469072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.468661    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5lg\" (UniqueName: \"kubernetes.io/projected/ba87d160-7551-4ea4-a296-ebf9155b5b5b-kube-api-access-gf5lg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:22.556503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.556411    5031 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:23:22.556503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.556473    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:23:22.570031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.570002    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-kube-api-access-wc7p6\" (UniqueName: \"kubernetes.io/projected/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-kube-api-access-wc7p6\") pod \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") "
Oct 30 22:23:22.570358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.570341    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-plugins-config-volume\") pod \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\" (UID: \"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e\") "
Oct 30 22:23:22.570707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.570684    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-plugins-config-volume" pod "62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" (UID: "62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:22.570890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.570877    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:22.572324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.572300    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-kube-api-access-wc7p6" pod "62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" (UID: "62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e"). InnerVolumeSpecName "kube-api-access-wc7p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:22.672183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:22.672138    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc7p6\" (UniqueName: \"kubernetes.io/projected/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e-kube-api-access-wc7p6\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:23.273674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.273573    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"
Oct 30 22:23:23.273674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.273593    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g" event={"ID":"62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e","Type":"ContainerDied","Data":"d4ceddef590bc426d992954abca17333c93a363caddc895931c3ce3ca9621a99"}
Oct 30 22:23:23.273951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.273696    5031 scope.go:122] "RemoveContainer" containerID="1c538834c1fd158068dc991e97fe09fcfec45174ca27b5a5586449155a727955"
Oct 30 22:23:23.275722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.275703    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"
Oct 30 22:23:23.275857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.275731    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k" event={"ID":"ba87d160-7551-4ea4-a296-ebf9155b5b5b","Type":"ContainerDied","Data":"278792cff09ad16329fb2928acaf20f0fbaa8f858dc7f662c5ebe1ada0f68b78"}
Oct 30 22:23:23.279053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.279024    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx" event={"ID":"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f","Type":"ContainerStarted","Data":"3a8a30f3605d22f020a0ed44f2be17ce63125d7e4fefa14cabd19f12fe0db852"}
Oct 30 22:23:23.279186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.279076    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx" event={"ID":"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f","Type":"ContainerStarted","Data":"d1f8bddbdf6ce8f3d11b601816bb3ec5bdb55571eebd3831719bf22dab4b3cc0"}
Oct 30 22:23:23.287516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.287486    5031 scope.go:122] "RemoveContainer" containerID="fafce537e2c070101c8fdd205cbc95c0949a89ef44dff543acac85c9eb28ce28"
Oct 30 22:23:23.458358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.458284    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx" podStartSLOduration=2.458264587 podStartE2EDuration="2.458264587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:23.452777132 +0000 UTC m=+2014.976919955" watchObservedRunningTime="2026-10-30 22:23:23.458264587 +0000 UTC m=+2014.982407354"
Oct 30 22:23:23.571645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.571555    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"]
Oct 30 22:23:23.575755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.575732    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k"]
Oct 30 22:23:23.588857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.588827    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:23:23.589387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.589187    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:23:23.590741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.590713    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:23:23.689055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.689028    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"]
Oct 30 22:23:23.695228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:23.695190    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g"]
Oct 30 22:23:24.289982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:24.289936    5031 generic.go:356] "Generic (PLEG): container finished" podID="f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" containerID="3a8a30f3605d22f020a0ed44f2be17ce63125d7e4fefa14cabd19f12fe0db852" exitCode=0
Oct 30 22:23:24.290234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:24.290035    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx" event={"ID":"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f","Type":"ContainerDied","Data":"3a8a30f3605d22f020a0ed44f2be17ce63125d7e4fefa14cabd19f12fe0db852"}
Oct 30 22:23:24.649178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:24.649133    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" path="/var/lib/kubelet/pods/62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e/volumes"
Oct 30 22:23:24.649768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:24.649743    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba87d160-7551-4ea4-a296-ebf9155b5b5b" path="/var/lib/kubelet/pods/ba87d160-7551-4ea4-a296-ebf9155b5b5b/volumes"
Oct 30 22:23:25.487398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.487371    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:25.516730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.516697    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-kube-api-access-bst9p\" (UniqueName: \"kubernetes.io/projected/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-kube-api-access-bst9p\") pod \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") "
Oct 30 22:23:25.516904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.516784    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-openshift-ca\") pod \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") "
Oct 30 22:23:25.517201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.517175    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-openshift-ca" pod "f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" (UID: "f8e750cc-2e81-444a-b2be-fac2ca9f2e5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:25.517448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.517428    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:25.519582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.519545    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-kube-api-access-bst9p" pod "f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" (UID: "f8e750cc-2e81-444a-b2be-fac2ca9f2e5f"). InnerVolumeSpecName "kube-api-access-bst9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:25.618184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.618146    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-xks-ca\") pod \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\" (UID: \"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f\") "
Oct 30 22:23:25.618714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.618694    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bst9p\" (UniqueName: \"kubernetes.io/projected/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-kube-api-access-bst9p\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:25.620850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.620824    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-xks-ca" pod "f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" (UID: "f8e750cc-2e81-444a-b2be-fac2ca9f2e5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:25.719898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:25.719858    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:26.311282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.311248    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx" event={"ID":"f8e750cc-2e81-444a-b2be-fac2ca9f2e5f","Type":"ContainerDied","Data":"d1f8bddbdf6ce8f3d11b601816bb3ec5bdb55571eebd3831719bf22dab4b3cc0"}
Oct 30 22:23:26.311492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.311304    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f8bddbdf6ce8f3d11b601816bb3ec5bdb55571eebd3831719bf22dab4b3cc0"
Oct 30 22:23:26.311492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.311303    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"
Oct 30 22:23:26.820253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.820186    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"]
Oct 30 22:23:26.820995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.820977    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" containerName="revoke-keys"
Oct 30 22:23:26.821066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.820996    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" containerName="revoke-keys"
Oct 30 22:23:26.821066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.821014    5031 cpu_manager.go:414] "containerMap: removing container" podUID="62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" containerName="payload-pre-processing"
Oct 30 22:23:26.821066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.821020    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" containerName="payload-pre-processing"
Oct 30 22:23:26.821066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.821056    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ba87d160-7551-4ea4-a296-ebf9155b5b5b" containerName="payload-processing"
Oct 30 22:23:26.821066 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.821061    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba87d160-7551-4ea4-a296-ebf9155b5b5b" containerName="payload-processing"
Oct 30 22:23:26.821347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.821172    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" containerName="revoke-keys"
Oct 30 22:23:26.821347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.821183    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c7f8d7-3827-4eb9-acde-d99f7c7b9e0e" containerName="payload-pre-processing"
Oct 30 22:23:26.821347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.821196    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba87d160-7551-4ea4-a296-ebf9155b5b5b" containerName="payload-processing"
Oct 30 22:23:26.824109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.824086    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.836409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.836381    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzltq\" (UniqueName: \"kubernetes.io/projected/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-kube-api-access-pzltq\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.836641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.836619    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.836715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.836687    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.882534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.882494    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:26.882534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.882519    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:26.882534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.882497    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:26.883682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.883656    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"]
Oct 30 22:23:26.928484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.928450    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:23:26.937963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.937930    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzltq\" (UniqueName: \"kubernetes.io/projected/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-kube-api-access-pzltq\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.938117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.938059    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.938117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.938097    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.938832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.938806    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:26.940799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:26.940762    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:27.001809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:27.001772    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzltq\" (UniqueName: \"kubernetes.io/projected/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-kube-api-access-pzltq\") pod \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:27.144053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:27.144015    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:27.547418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:27.547285    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"]
Oct 30 22:23:27.547757 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:23:27.547728    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2b25bd_4dc9_4fb1_9537_e372ce7afcf6.slice/crio-61ec7f1c3ce67672c45dcc83c52cac66dce83119433461f7d4851556f78a6b63 WatchSource:0}: Error finding container 61ec7f1c3ce67672c45dcc83c52cac66dce83119433461f7d4851556f78a6b63: Status 404 returned error can't find the container with id 61ec7f1c3ce67672c45dcc83c52cac66dce83119433461f7d4851556f78a6b63
Oct 30 22:23:27.974569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:27.974533    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:28.336859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:28.336806    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s" event={"ID":"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6","Type":"ContainerStarted","Data":"22ed896b54074f5ed23d85ca9279d0e4c84143bbebc71d8d82e0a5c737054fbc"}
Oct 30 22:23:28.337137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:28.337110    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s" event={"ID":"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6","Type":"ContainerStarted","Data":"61ec7f1c3ce67672c45dcc83c52cac66dce83119433461f7d4851556f78a6b63"}
Oct 30 22:23:28.459454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:28.459402    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s" podStartSLOduration=2.459387606 podStartE2EDuration="2.459387606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:28.456635167 +0000 UTC m=+2019.980777933" watchObservedRunningTime="2026-10-30 22:23:28.459387606 +0000 UTC m=+2019.983530445"
Oct 30 22:23:29.348688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:29.348652    5031 generic.go:356] "Generic (PLEG): container finished" podID="0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" containerID="22ed896b54074f5ed23d85ca9279d0e4c84143bbebc71d8d82e0a5c737054fbc" exitCode=0
Oct 30 22:23:29.349260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:29.348686    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s" event={"ID":"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6","Type":"ContainerDied","Data":"22ed896b54074f5ed23d85ca9279d0e4c84143bbebc71d8d82e0a5c737054fbc"}
Oct 30 22:23:30.548289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.548263    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:30.597421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.597385    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-openshift-ca\") pod \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") "
Oct 30 22:23:30.597583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.597531    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-xks-ca\") pod \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") "
Oct 30 22:23:30.597692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.597674    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-kube-api-access-pzltq\" (UniqueName: \"kubernetes.io/projected/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-kube-api-access-pzltq\") pod \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\" (UID: \"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6\") "
Oct 30 22:23:30.597818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.597799    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-openshift-ca" pod "0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" (UID: "0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:30.598348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.598323    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.599912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.599881    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-xks-ca" pod "0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" (UID: "0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:30.600006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.599936    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-kube-api-access-pzltq" pod "0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" (UID: "0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6"). InnerVolumeSpecName "kube-api-access-pzltq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:30.700680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.700640    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.700680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:30.700681    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzltq\" (UniqueName: \"kubernetes.io/projected/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6-kube-api-access-pzltq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:31.368905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:31.368864    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s" event={"ID":"0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6","Type":"ContainerDied","Data":"61ec7f1c3ce67672c45dcc83c52cac66dce83119433461f7d4851556f78a6b63"}
Oct 30 22:23:31.369074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:31.368929    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ec7f1c3ce67672c45dcc83c52cac66dce83119433461f7d4851556f78a6b63"
Oct 30 22:23:31.369074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:31.368879    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:23:33.456040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:33.455984    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:34.366703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:34.366667    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:23:37.209381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:37.209337    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"]
Oct 30 22:23:37.325120 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:37.325085    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"]
Oct 30 22:23:37.670511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:37.670427    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:37.892718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:37.892680    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:38.965841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:38.965809    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:42.210453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.210409    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt" podUID="182013cc-390b-477e-8a92-08b67bf64ab8" containerName="payload-processing" containerID="cri-o://9ff74aac9a873785062a27cad3b8ad8272f56a308e575b3978c58ebc279343a8" gracePeriod=25
Oct 30 22:23:42.325833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.325790    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv" podUID="7d1cf0e4-a200-4525-aea5-ab968f265226" containerName="payload-pre-processing" containerID="cri-o://bba9c8aa2147e278695fac69ecd433e8a52653ab125525fec4d79a8d1b9cb5a7" gracePeriod=25
Oct 30 22:23:42.483336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.483298    5031 generic.go:356] "Generic (PLEG): container finished" podID="182013cc-390b-477e-8a92-08b67bf64ab8" containerID="9ff74aac9a873785062a27cad3b8ad8272f56a308e575b3978c58ebc279343a8" exitCode=0
Oct 30 22:23:42.483522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.483492    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt" event={"ID":"182013cc-390b-477e-8a92-08b67bf64ab8","Type":"ContainerDied","Data":"9ff74aac9a873785062a27cad3b8ad8272f56a308e575b3978c58ebc279343a8"}
Oct 30 22:23:42.485717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.485693    5031 generic.go:356] "Generic (PLEG): container finished" podID="7d1cf0e4-a200-4525-aea5-ab968f265226" containerID="bba9c8aa2147e278695fac69ecd433e8a52653ab125525fec4d79a8d1b9cb5a7" exitCode=0
Oct 30 22:23:42.485840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.485764    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv" event={"ID":"7d1cf0e4-a200-4525-aea5-ab968f265226","Type":"ContainerDied","Data":"bba9c8aa2147e278695fac69ecd433e8a52653ab125525fec4d79a8d1b9cb5a7"}
Oct 30 22:23:42.520387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.520354    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:23:42.622076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.622046    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182013cc-390b-477e-8a92-08b67bf64ab8-kube-api-access-t8lll\" (UniqueName: \"kubernetes.io/projected/182013cc-390b-477e-8a92-08b67bf64ab8-kube-api-access-t8lll\") pod \"182013cc-390b-477e-8a92-08b67bf64ab8\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") "
Oct 30 22:23:42.622464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.622442    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182013cc-390b-477e-8a92-08b67bf64ab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182013cc-390b-477e-8a92-08b67bf64ab8-plugins-config-volume\") pod \"182013cc-390b-477e-8a92-08b67bf64ab8\" (UID: \"182013cc-390b-477e-8a92-08b67bf64ab8\") "
Oct 30 22:23:42.622866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.622837    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182013cc-390b-477e-8a92-08b67bf64ab8-plugins-config-volume" pod "182013cc-390b-477e-8a92-08b67bf64ab8" (UID: "182013cc-390b-477e-8a92-08b67bf64ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:42.623171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.623148    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182013cc-390b-477e-8a92-08b67bf64ab8-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:42.625068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.625043    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182013cc-390b-477e-8a92-08b67bf64ab8-kube-api-access-t8lll" pod "182013cc-390b-477e-8a92-08b67bf64ab8" (UID: "182013cc-390b-477e-8a92-08b67bf64ab8"). InnerVolumeSpecName "kube-api-access-t8lll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:42.640604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.640581    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:23:42.724250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.724190    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1cf0e4-a200-4525-aea5-ab968f265226-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1cf0e4-a200-4525-aea5-ab968f265226-plugins-config-volume\") pod \"7d1cf0e4-a200-4525-aea5-ab968f265226\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") "
Oct 30 22:23:42.724457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.724322    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1cf0e4-a200-4525-aea5-ab968f265226-kube-api-access-2dwnq\" (UniqueName: \"kubernetes.io/projected/7d1cf0e4-a200-4525-aea5-ab968f265226-kube-api-access-2dwnq\") pod \"7d1cf0e4-a200-4525-aea5-ab968f265226\" (UID: \"7d1cf0e4-a200-4525-aea5-ab968f265226\") "
Oct 30 22:23:42.724597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.724571    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1cf0e4-a200-4525-aea5-ab968f265226-plugins-config-volume" pod "7d1cf0e4-a200-4525-aea5-ab968f265226" (UID: "7d1cf0e4-a200-4525-aea5-ab968f265226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:42.725097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.725077    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8lll\" (UniqueName: \"kubernetes.io/projected/182013cc-390b-477e-8a92-08b67bf64ab8-kube-api-access-t8lll\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:42.725225 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.725100    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1cf0e4-a200-4525-aea5-ab968f265226-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:42.726861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.726808    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1cf0e4-a200-4525-aea5-ab968f265226-kube-api-access-2dwnq" pod "7d1cf0e4-a200-4525-aea5-ab968f265226" (UID: "7d1cf0e4-a200-4525-aea5-ab968f265226"). InnerVolumeSpecName "kube-api-access-2dwnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:42.827054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:42.827014    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwnq\" (UniqueName: \"kubernetes.io/projected/7d1cf0e4-a200-4525-aea5-ab968f265226-kube-api-access-2dwnq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:43.495774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.495688    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"
Oct 30 22:23:43.495774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.495718    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt" event={"ID":"182013cc-390b-477e-8a92-08b67bf64ab8","Type":"ContainerDied","Data":"7cedb3770e3ee3ed595749cbb6cf30596ebbbbdd05a83255c996bbf36300b4bb"}
Oct 30 22:23:43.496327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.495797    5031 scope.go:122] "RemoveContainer" containerID="9ff74aac9a873785062a27cad3b8ad8272f56a308e575b3978c58ebc279343a8"
Oct 30 22:23:43.497798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.497506    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"
Oct 30 22:23:43.497798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.497539    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv" event={"ID":"7d1cf0e4-a200-4525-aea5-ab968f265226","Type":"ContainerDied","Data":"094e9a031ddc6aff6ec6cbedf67488650d11b196f727777b795e376f36e83267"}
Oct 30 22:23:43.510633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.510611    5031 scope.go:122] "RemoveContainer" containerID="bba9c8aa2147e278695fac69ecd433e8a52653ab125525fec4d79a8d1b9cb5a7"
Oct 30 22:23:43.669146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.669107    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"]
Oct 30 22:23:43.719769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.719733    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv"]
Oct 30 22:23:43.846277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.846225    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"]
Oct 30 22:23:43.850387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:43.850356    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt"]
Oct 30 22:23:44.127530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:44.127437    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:44.648644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:44.648610    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182013cc-390b-477e-8a92-08b67bf64ab8" path="/var/lib/kubelet/pods/182013cc-390b-477e-8a92-08b67bf64ab8/volumes"
Oct 30 22:23:44.649169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:44.649153    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1cf0e4-a200-4525-aea5-ab968f265226" path="/var/lib/kubelet/pods/7d1cf0e4-a200-4525-aea5-ab968f265226/volumes"
Oct 30 22:23:49.278247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:49.278204    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:50.546539 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:23:50.546499    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:23:50.559251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:50.559199    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:54.383418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.383377    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:54.745332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.745247    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"]
Oct 30 22:23:54.746354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746331    5031 cpu_manager.go:414] "containerMap: removing container" podUID="182013cc-390b-477e-8a92-08b67bf64ab8" containerName="payload-processing"
Oct 30 22:23:54.746411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746357    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182013cc-390b-477e-8a92-08b67bf64ab8" containerName="payload-processing"
Oct 30 22:23:54.746411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746397    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" containerName="revoke-keys"
Oct 30 22:23:54.746411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746404    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" containerName="revoke-keys"
Oct 30 22:23:54.746509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746424    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7d1cf0e4-a200-4525-aea5-ab968f265226" containerName="payload-pre-processing"
Oct 30 22:23:54.746509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746433    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1cf0e4-a200-4525-aea5-ab968f265226" containerName="payload-pre-processing"
Oct 30 22:23:54.746905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746638    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="182013cc-390b-477e-8a92-08b67bf64ab8" containerName="payload-processing"
Oct 30 22:23:54.746905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746656    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" containerName="revoke-keys"
Oct 30 22:23:54.746905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.746669    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1cf0e4-a200-4525-aea5-ab968f265226" containerName="payload-pre-processing"
Oct 30 22:23:54.753121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.753095    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:54.807000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.806967    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"]
Oct 30 22:23:54.807895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.807872    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:54.807995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.807921    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:54.807995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.807972    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:23:54.808069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.808000    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:54.923181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.923146    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:54.923181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.923183    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:54.923473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:54.923231    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdk6\" (UniqueName: \"kubernetes.io/projected/82167de4-fc98-4ec1-8a55-74ff2291b322-kube-api-access-pvdk6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.025058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.024965    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdk6\" (UniqueName: \"kubernetes.io/projected/82167de4-fc98-4ec1-8a55-74ff2291b322-kube-api-access-pvdk6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.025323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.025303    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.025438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.025345    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.025917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.025894    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.027888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.027868    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.088459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.088424    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdk6\" (UniqueName: \"kubernetes.io/projected/82167de4-fc98-4ec1-8a55-74ff2291b322-kube-api-access-pvdk6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.367491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.367455    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:55.759082 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:23:55.759051    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82167de4_fc98_4ec1_8a55_74ff2291b322.slice/crio-653d1963b0c8e29cbe84aca6fa76d4bdf2eda0140decb7c72273beaf1aaa4507 WatchSource:0}: Error finding container 653d1963b0c8e29cbe84aca6fa76d4bdf2eda0140decb7c72273beaf1aaa4507: Status 404 returned error can't find the container with id 653d1963b0c8e29cbe84aca6fa76d4bdf2eda0140decb7c72273beaf1aaa4507
Oct 30 22:23:55.760085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:55.760035    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"]
Oct 30 22:23:56.658405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:56.658361    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q" event={"ID":"82167de4-fc98-4ec1-8a55-74ff2291b322","Type":"ContainerStarted","Data":"4daf85a01e90f53500e578f2f45dab5fb3af1b335b274914f2a42c4f061d8ce5"}
Oct 30 22:23:56.658698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:56.658677    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q" event={"ID":"82167de4-fc98-4ec1-8a55-74ff2291b322","Type":"ContainerStarted","Data":"653d1963b0c8e29cbe84aca6fa76d4bdf2eda0140decb7c72273beaf1aaa4507"}
Oct 30 22:23:56.824984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:56.824921    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q" podStartSLOduration=2.824901017 podStartE2EDuration="2.824901017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:56.818954046 +0000 UTC m=+2048.343096835" watchObservedRunningTime="2026-10-30 22:23:56.824901017 +0000 UTC m=+2048.349043784"
Oct 30 22:23:57.669437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:57.669397    5031 generic.go:356] "Generic (PLEG): container finished" podID="82167de4-fc98-4ec1-8a55-74ff2291b322" containerID="4daf85a01e90f53500e578f2f45dab5fb3af1b335b274914f2a42c4f061d8ce5" exitCode=0
Oct 30 22:23:57.669641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:57.669491    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q" event={"ID":"82167de4-fc98-4ec1-8a55-74ff2291b322","Type":"ContainerDied","Data":"4daf85a01e90f53500e578f2f45dab5fb3af1b335b274914f2a42c4f061d8ce5"}
Oct 30 22:23:58.869103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.869077    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:58.982150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.982068    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-xks-ca\") pod \"82167de4-fc98-4ec1-8a55-74ff2291b322\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") "
Oct 30 22:23:58.982351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.982318    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82167de4-fc98-4ec1-8a55-74ff2291b322-kube-api-access-pvdk6\" (UniqueName: \"kubernetes.io/projected/82167de4-fc98-4ec1-8a55-74ff2291b322-kube-api-access-pvdk6\") pod \"82167de4-fc98-4ec1-8a55-74ff2291b322\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") "
Oct 30 22:23:58.982397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.982384    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-openshift-ca\") pod \"82167de4-fc98-4ec1-8a55-74ff2291b322\" (UID: \"82167de4-fc98-4ec1-8a55-74ff2291b322\") "
Oct 30 22:23:58.982767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.982726    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-openshift-ca" pod "82167de4-fc98-4ec1-8a55-74ff2291b322" (UID: "82167de4-fc98-4ec1-8a55-74ff2291b322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:58.983062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.982997    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:58.984528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.984501    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-xks-ca" pod "82167de4-fc98-4ec1-8a55-74ff2291b322" (UID: "82167de4-fc98-4ec1-8a55-74ff2291b322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:58.984641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:58.984609    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82167de4-fc98-4ec1-8a55-74ff2291b322-kube-api-access-pvdk6" pod "82167de4-fc98-4ec1-8a55-74ff2291b322" (UID: "82167de4-fc98-4ec1-8a55-74ff2291b322"). InnerVolumeSpecName "kube-api-access-pvdk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:59.084310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:59.084263    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdk6\" (UniqueName: \"kubernetes.io/projected/82167de4-fc98-4ec1-8a55-74ff2291b322-kube-api-access-pvdk6\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:59.084310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:59.084301    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82167de4-fc98-4ec1-8a55-74ff2291b322-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:59.687935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:59.687899    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"
Oct 30 22:23:59.687935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:59.687921    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q" event={"ID":"82167de4-fc98-4ec1-8a55-74ff2291b322","Type":"ContainerDied","Data":"653d1963b0c8e29cbe84aca6fa76d4bdf2eda0140decb7c72273beaf1aaa4507"}
Oct 30 22:23:59.688172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:59.687968    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="653d1963b0c8e29cbe84aca6fa76d4bdf2eda0140decb7c72273beaf1aaa4507"
Oct 30 22:23:59.923468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:23:59.923432    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:00.161420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:00.161387    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:05.689232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:05.681463    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:05.970429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:05.970344    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:10.271517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:10.271482    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.084988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:12.084951    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.326798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:12.326743    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.550770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:12.550673    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.679900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:12.679857    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"]
Oct 30 22:24:12.730152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:12.730115    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9b614c-qsn8q"]
Oct 30 22:24:13.222973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:13.222931    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:14.651293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:14.651253    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82167de4-fc98-4ec1-8a55-74ff2291b322" path="/var/lib/kubelet/pods/82167de4-fc98-4ec1-8a55-74ff2291b322/volumes"
Oct 30 22:24:17.172035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.171996    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"]
Oct 30 22:24:17.172729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.172713    5031 cpu_manager.go:414] "containerMap: removing container" podUID="82167de4-fc98-4ec1-8a55-74ff2291b322" containerName="revoke-keys"
Oct 30 22:24:17.172778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.172730    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82167de4-fc98-4ec1-8a55-74ff2291b322" containerName="revoke-keys"
Oct 30 22:24:17.172894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.172883    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="82167de4-fc98-4ec1-8a55-74ff2291b322" containerName="revoke-keys"
Oct 30 22:24:17.177047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.177021    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.243416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.243381    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"]
Oct 30 22:24:17.288454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.288402    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:17.288454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.288402    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:17.288454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.288409    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:17.288717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.288520    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:24:17.297877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.297847    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.298248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.298225    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.298337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.298316    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tm5\" (UniqueName: \"kubernetes.io/projected/4680090e-d503-49dc-8c25-4abdfcb7a2db-kube-api-access-c9tm5\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.399992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.399948    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.399992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.399996    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tm5\" (UniqueName: \"kubernetes.io/projected/4680090e-d503-49dc-8c25-4abdfcb7a2db-kube-api-access-c9tm5\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.400291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.400085    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.400780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.400754    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.402591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.402570    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.469115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.469024    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tm5\" (UniqueName: \"kubernetes.io/projected/4680090e-d503-49dc-8c25-4abdfcb7a2db-kube-api-access-c9tm5\") pod \"maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.493683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.493654    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:17.856796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:17.856766    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:18.091811 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:24:18.091759    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4680090e_d503_49dc_8c25_4abdfcb7a2db.slice/crio-05abf17d2339d00866d639f690c67fa94c23f28cebba06b34ebe041db1c51df1 WatchSource:0}: Error finding container 05abf17d2339d00866d639f690c67fa94c23f28cebba06b34ebe041db1c51df1: Status 404 returned error can't find the container with id 05abf17d2339d00866d639f690c67fa94c23f28cebba06b34ebe041db1c51df1
Oct 30 22:24:18.092348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:18.092323    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"]
Oct 30 22:24:18.922112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:18.922069    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb" event={"ID":"4680090e-d503-49dc-8c25-4abdfcb7a2db","Type":"ContainerStarted","Data":"aa6d22555e00d33f6c1196289ebd542ff607bc686e5839c266f571e17367c041"}
Oct 30 22:24:18.922618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:18.922142    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb" event={"ID":"4680090e-d503-49dc-8c25-4abdfcb7a2db","Type":"ContainerStarted","Data":"05abf17d2339d00866d639f690c67fa94c23f28cebba06b34ebe041db1c51df1"}
Oct 30 22:24:19.047451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:19.047396    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb" podStartSLOduration=2.047380646 podStartE2EDuration="2.047380646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:19.044593461 +0000 UTC m=+2070.568736236" watchObservedRunningTime="2026-10-30 22:24:19.047380646 +0000 UTC m=+2070.571523490"
Oct 30 22:24:19.933862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:19.933821    5031 generic.go:356] "Generic (PLEG): container finished" podID="4680090e-d503-49dc-8c25-4abdfcb7a2db" containerID="aa6d22555e00d33f6c1196289ebd542ff607bc686e5839c266f571e17367c041" exitCode=0
Oct 30 22:24:19.934427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:19.933910    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb" event={"ID":"4680090e-d503-49dc-8c25-4abdfcb7a2db","Type":"ContainerDied","Data":"aa6d22555e00d33f6c1196289ebd542ff607bc686e5839c266f571e17367c041"}
Oct 30 22:24:20.544728 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:24:20.544686    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:24:21.018021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.017985    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:21.167059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.167030    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:21.294223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.294098    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-openshift-ca\") pod \"4680090e-d503-49dc-8c25-4abdfcb7a2db\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") "
Oct 30 22:24:21.294919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.294892    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4680090e-d503-49dc-8c25-4abdfcb7a2db-kube-api-access-c9tm5\" (UniqueName: \"kubernetes.io/projected/4680090e-d503-49dc-8c25-4abdfcb7a2db-kube-api-access-c9tm5\") pod \"4680090e-d503-49dc-8c25-4abdfcb7a2db\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") "
Oct 30 22:24:21.297273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.294511    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-openshift-ca" pod "4680090e-d503-49dc-8c25-4abdfcb7a2db" (UID: "4680090e-d503-49dc-8c25-4abdfcb7a2db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:21.297438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.294568    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"]
Oct 30 22:24:21.299357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.297542    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-xks-ca\") pod \"4680090e-d503-49dc-8c25-4abdfcb7a2db\" (UID: \"4680090e-d503-49dc-8c25-4abdfcb7a2db\") "
Oct 30 22:24:21.299357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.299320    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:21.299855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.299784    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy" containerID="cri-o://9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13" gracePeriod=30
Oct 30 22:24:21.301004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.300437    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4680090e-d503-49dc-8c25-4abdfcb7a2db-kube-api-access-c9tm5" pod "4680090e-d503-49dc-8c25-4abdfcb7a2db" (UID: "4680090e-d503-49dc-8c25-4abdfcb7a2db"). InnerVolumeSpecName "kube-api-access-c9tm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:21.301004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.300522    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-xks-ca" pod "4680090e-d503-49dc-8c25-4abdfcb7a2db" (UID: "4680090e-d503-49dc-8c25-4abdfcb7a2db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:21.318159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.318108    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:21.400871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.400831    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tm5\" (UniqueName: \"kubernetes.io/projected/4680090e-d503-49dc-8c25-4abdfcb7a2db-kube-api-access-c9tm5\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:21.400871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.400875    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680090e-d503-49dc-8c25-4abdfcb7a2db-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:21.953334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.953300    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"
Oct 30 22:24:21.953538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.953339    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb" event={"ID":"4680090e-d503-49dc-8c25-4abdfcb7a2db","Type":"ContainerDied","Data":"05abf17d2339d00866d639f690c67fa94c23f28cebba06b34ebe041db1c51df1"}
Oct 30 22:24:21.953538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:21.953423    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05abf17d2339d00866d639f690c67fa94c23f28cebba06b34ebe041db1c51df1"
Oct 30 22:24:22.158561 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.158525    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"]
Oct 30 22:24:22.159542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.159517    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4680090e-d503-49dc-8c25-4abdfcb7a2db" containerName="revoke-keys"
Oct 30 22:24:22.159542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.159541    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4680090e-d503-49dc-8c25-4abdfcb7a2db" containerName="revoke-keys"
Oct 30 22:24:22.159754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.159719    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4680090e-d503-49dc-8c25-4abdfcb7a2db" containerName="revoke-keys"
Oct 30 22:24:22.163592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.163568    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.220055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.219950    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:22.220055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.219951    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:22.220055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.219968    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:22.220437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.220425    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:24:22.224096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.224066    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"]
Oct 30 22:24:22.322054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.322014    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.322284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.322068    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8kc7\" (UniqueName: \"kubernetes.io/projected/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-kube-api-access-g8kc7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.322342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.322282    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.424096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.424047    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.424338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.424314    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.424401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.424384    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8kc7\" (UniqueName: \"kubernetes.io/projected/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-kube-api-access-g8kc7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.424934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.424898    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.427293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.427266    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.490395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.490305    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8kc7\" (UniqueName: \"kubernetes.io/projected/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-kube-api-access-g8kc7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:22.781234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:22.781110    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:23.190968 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:24:23.190919    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaac269_12f8_4ca2_a1d1_b3ab41f3a4bc.slice/crio-05506ffeea32dd3a4a63bee89e7902c986f1de18932392338a9a04b2301042ef WatchSource:0}: Error finding container 05506ffeea32dd3a4a63bee89e7902c986f1de18932392338a9a04b2301042ef: Status 404 returned error can't find the container with id 05506ffeea32dd3a4a63bee89e7902c986f1de18932392338a9a04b2301042ef
Oct 30 22:24:23.191481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:23.191081    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"]
Oct 30 22:24:23.887363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:23.887331    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:23.981674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:23.981636    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc" event={"ID":"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc","Type":"ContainerStarted","Data":"fe9978f66614a11eda1f7891241003e23acc2dd0ff6f55b9ac11dc8614c580c2"}
Oct 30 22:24:23.981870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:23.981692    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc" event={"ID":"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc","Type":"ContainerStarted","Data":"05506ffeea32dd3a4a63bee89e7902c986f1de18932392338a9a04b2301042ef"}
Oct 30 22:24:24.106253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:24.106176    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc" podStartSLOduration=2.106159799 podStartE2EDuration="2.106159799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:24.102642329 +0000 UTC m=+2075.626785104" watchObservedRunningTime="2026-10-30 22:24:24.106159799 +0000 UTC m=+2075.630302566"
Oct 30 22:24:24.991700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:24.991662    5031 generic.go:356] "Generic (PLEG): container finished" podID="ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" containerID="fe9978f66614a11eda1f7891241003e23acc2dd0ff6f55b9ac11dc8614c580c2" exitCode=0
Oct 30 22:24:24.992200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:24.991750    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc" event={"ID":"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc","Type":"ContainerDied","Data":"fe9978f66614a11eda1f7891241003e23acc2dd0ff6f55b9ac11dc8614c580c2"}
Oct 30 22:24:26.175983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.175956    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:26.281832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.281748    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-openshift-ca\") pod \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") "
Oct 30 22:24:26.282014 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.281840    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-kube-api-access-g8kc7\" (UniqueName: \"kubernetes.io/projected/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-kube-api-access-g8kc7\") pod \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") "
Oct 30 22:24:26.282014 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.281889    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-xks-ca\") pod \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\" (UID: \"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc\") "
Oct 30 22:24:26.282197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.282175    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-openshift-ca" pod "ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" (UID: "ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:26.282648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.282629    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.284158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.284136    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-xks-ca" pod "ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" (UID: "ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:26.284240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.284201    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-kube-api-access-g8kc7" pod "ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" (UID: "ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc"). InnerVolumeSpecName "kube-api-access-g8kc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:26.383727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.383692    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8kc7\" (UniqueName: \"kubernetes.io/projected/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-kube-api-access-g8kc7\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.383727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.383724    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.576688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.576661    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:24:26.688596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.688563    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istiod-ca-cert\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.688761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.688669    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-certs\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.688761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.688707    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-data\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.688761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.688753    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-credential-socket\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.688895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.688792    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-kube-api-access-b8lkj\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-kube-api-access-b8lkj\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.688955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.688945    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-socket\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.689075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.688974    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-podinfo\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.689075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689000    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-data" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:26.689075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689019    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-certs" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:26.689075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689045    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istiod-ca-cert" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:26.689356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689126    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-envoy\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.689356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689165    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-token\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-token\") pod \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\" (UID: \"7cb3203b-ff2d-4c68-afa6-e19c232d70a0\") "
Oct 30 22:24:26.689356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689251    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-socket" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:26.689520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689423    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-credential-socket" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:26.689862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689843    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.689976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689865    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.689976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689881    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.689976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689894    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.689976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.689906    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.691489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.691465    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-podinfo" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:26.691622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.691573    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-envoy" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:26.691950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.691928    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-kube-api-access-b8lkj" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "kube-api-access-b8lkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:26.692041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.691958    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-token" pod "7cb3203b-ff2d-4c68-afa6-e19c232d70a0" (UID: "7cb3203b-ff2d-4c68-afa6-e19c232d70a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:26.791417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.791376    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.791417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.791416    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.791659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.791437    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8lkj\" (UniqueName: \"kubernetes.io/projected/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-kube-api-access-b8lkj\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.791659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:26.791455    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb3203b-ff2d-4c68-afa6-e19c232d70a0-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:27.009626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.009585    5031 generic.go:356] "Generic (PLEG): container finished" podID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerID="9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13" exitCode=0
Oct 30 22:24:27.009833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.009634    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"
Oct 30 22:24:27.009833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.009672    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" event={"ID":"7cb3203b-ff2d-4c68-afa6-e19c232d70a0","Type":"ContainerDied","Data":"9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13"}
Oct 30 22:24:27.009833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.009731    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd" event={"ID":"7cb3203b-ff2d-4c68-afa6-e19c232d70a0","Type":"ContainerDied","Data":"0e6b0699801562d9ffd5d9252efe5e72a299e7a9186dcd8c8ee3b3f6045c65c2"}
Oct 30 22:24:27.009833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.009761    5031 scope.go:122] "RemoveContainer" containerID="9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13"
Oct 30 22:24:27.011460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.011433    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"
Oct 30 22:24:27.011673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.011562    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc" event={"ID":"ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc","Type":"ContainerDied","Data":"05506ffeea32dd3a4a63bee89e7902c986f1de18932392338a9a04b2301042ef"}
Oct 30 22:24:27.011805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.011699    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05506ffeea32dd3a4a63bee89e7902c986f1de18932392338a9a04b2301042ef"
Oct 30 22:24:27.022756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.022736    5031 scope.go:122] "RemoveContainer" containerID="492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23"
Oct 30 22:24:27.033938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.033914    5031 scope.go:122] "RemoveContainer" containerID="9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13"
Oct 30 22:24:27.034233 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:24:27.034197    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13\": container with ID starting with 9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13 not found: ID does not exist" containerID="9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13"
Oct 30 22:24:27.034316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.034244    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13"} err="failed to get container status \"9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13\": rpc error: code = NotFound desc = could not find container \"9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13\": container with ID starting with 9c71243366974dc75c4b3bc2ce7d75840d2b73e00ad48f689cb5c499c2a9de13 not found: ID does not exist"
Oct 30 22:24:27.034316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.034265    5031 scope.go:122] "RemoveContainer" containerID="492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23"
Oct 30 22:24:27.034539 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:24:27.034524    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23\": container with ID starting with 492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23 not found: ID does not exist" containerID="492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23"
Oct 30 22:24:27.034588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.034544    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23"} err="failed to get container status \"492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23\": rpc error: code = NotFound desc = could not find container \"492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23\": container with ID starting with 492d4ff2e578bc10bb1c4e309b85c9f7c47dcf0554bf68c847bcd2c7c4f79a23 not found: ID does not exist"
Oct 30 22:24:27.185651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.185614    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"]
Oct 30 22:24:27.236269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:27.236199    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd"]
Oct 30 22:24:28.029237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:28.028452    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:28.283028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:28.282946    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:28.647920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:28.647880    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" path="/var/lib/kubelet/pods/7cb3203b-ff2d-4c68-afa6-e19c232d70a0/volumes"
Oct 30 22:24:29.125234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:29.117763    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:33.566315 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:33.563813    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:33.820328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:33.820232    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:34.052376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:34.052339    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.498472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:35.498436    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.785130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:35.785028    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:36.013068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:36.013027    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"]
Oct 30 22:24:36.018479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:36.018449    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:36.061099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:36.061019    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-41e4de-9fh5b"]
Oct 30 22:24:36.461762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:36.461721    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:36.647950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:36.647908    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602bf24f-1360-4ff4-b0d3-6ba1e791ecdb" path="/var/lib/kubelet/pods/602bf24f-1360-4ff4-b0d3-6ba1e791ecdb/volumes"
Oct 30 22:24:37.066549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:37.066513    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:37.306254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:37.306202    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:40.605474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:40.605432    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:41.775888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:41.775852    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.194498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:42.194459    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.468894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:42.468809    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:42.648735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:42.648700    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"]
Oct 30 22:24:42.689286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:42.689252    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81f37a-vspsq"]
Oct 30 22:24:43.088146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:43.088104    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.343298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:43.337794    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.576907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:43.576870    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.815168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:43.815063    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.975824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:43.975779    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"]
Oct 30 22:24:44.053657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.053608    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-18edc4-n7stb"]
Oct 30 22:24:44.488315 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.488284    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:44.648682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.648637    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4680090e-d503-49dc-8c25-4abdfcb7a2db" path="/var/lib/kubelet/pods/4680090e-d503-49dc-8c25-4abdfcb7a2db/volumes"
Oct 30 22:24:44.649152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.649136    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1821963-2caf-4b69-818b-a4cff2901fd9" path="/var/lib/kubelet/pods/b1821963-2caf-4b69-818b-a4cff2901fd9/volumes"
Oct 30 22:24:44.726747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.726717    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:44.868838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.868804    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"]
Oct 30 22:24:44.869124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.869099    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy" containerID="cri-o://d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f" gracePeriod=30
Oct 30 22:24:44.889027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.889000    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:24:44.988863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:44.988740    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.233987 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.233899    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.407126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.407091    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"]
Oct 30 22:24:45.411129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.411097    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b55a43-n54kw"]
Oct 30 22:24:45.484596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.484511    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.885942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.885824    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"]
Oct 30 22:24:45.887664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.887633    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy"
Oct 30 22:24:45.887814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.887789    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy"
Oct 30 22:24:45.887958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.887946    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy"
Oct 30 22:24:45.888048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.888037    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy"
Oct 30 22:24:45.888178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.888166    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" containerName="revoke-keys"
Oct 30 22:24:45.888285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.888273    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" containerName="revoke-keys"
Oct 30 22:24:45.888588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.888573    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" containerName="revoke-keys"
Oct 30 22:24:45.888703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.888692    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy"
Oct 30 22:24:45.888799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.888787    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb3203b-ff2d-4c68-afa6-e19c232d70a0" containerName="istio-proxy"
Oct 30 22:24:45.903923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.903894    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:45.937313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.937286    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.945366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.944392    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"]
Oct 30 22:24:45.960917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.960886    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:45.964692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:45.964630    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:24:46.022530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.022498    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:46.022530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.022498    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:46.068153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.068115    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqh8x\" (UniqueName: \"kubernetes.io/projected/09e359e3-5dd3-429f-9d5a-1d75e5d23473-kube-api-access-pqh8x\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.068372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.068273    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.068453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.068431    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.178286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.174546    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.178286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.174701    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.178286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.174847    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqh8x\" (UniqueName: \"kubernetes.io/projected/09e359e3-5dd3-429f-9d5a-1d75e5d23473-kube-api-access-pqh8x\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.178286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.177238    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.180488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.180461    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.249226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.247896    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqh8x\" (UniqueName: \"kubernetes.io/projected/09e359e3-5dd3-429f-9d5a-1d75e5d23473-kube-api-access-pqh8x\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.265906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.265870    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:46.647769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.647732    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c4acc6-714a-464c-bea6-6b9733903a62" path="/var/lib/kubelet/pods/d3c4acc6-714a-464c-bea6-6b9733903a62/volumes"
Oct 30 22:24:46.657049 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:24:46.657019    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e359e3_5dd3_429f_9d5a_1d75e5d23473.slice/crio-40602462dd421e7fab5e6e6e40ef26b0ef57e5c0de740add2edbae28ee5d99a1 WatchSource:0}: Error finding container 40602462dd421e7fab5e6e6e40ef26b0ef57e5c0de740add2edbae28ee5d99a1: Status 404 returned error can't find the container with id 40602462dd421e7fab5e6e6e40ef26b0ef57e5c0de740add2edbae28ee5d99a1
Oct 30 22:24:46.658780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:46.658647    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"]
Oct 30 22:24:47.235311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:47.235254    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt" event={"ID":"09e359e3-5dd3-429f-9d5a-1d75e5d23473","Type":"ContainerStarted","Data":"342663b47b7bdc36b2b22dacb388d1a5b010f0ec93c75a68ca0c71d8bd885e45"}
Oct 30 22:24:47.235803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:47.235356    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt" event={"ID":"09e359e3-5dd3-429f-9d5a-1d75e5d23473","Type":"ContainerStarted","Data":"40602462dd421e7fab5e6e6e40ef26b0ef57e5c0de740add2edbae28ee5d99a1"}
Oct 30 22:24:47.392678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:47.392615    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt" podStartSLOduration=2.392596872 podStartE2EDuration="2.392596872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:47.388455705 +0000 UTC m=+2098.912598479" watchObservedRunningTime="2026-10-30 22:24:47.392596872 +0000 UTC m=+2098.916739639"
Oct 30 22:24:48.246067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:48.246029    5031 generic.go:356] "Generic (PLEG): container finished" podID="09e359e3-5dd3-429f-9d5a-1d75e5d23473" containerID="342663b47b7bdc36b2b22dacb388d1a5b010f0ec93c75a68ca0c71d8bd885e45" exitCode=0
Oct 30 22:24:48.246516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:48.246119    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt" event={"ID":"09e359e3-5dd3-429f-9d5a-1d75e5d23473","Type":"ContainerDied","Data":"342663b47b7bdc36b2b22dacb388d1a5b010f0ec93c75a68ca0c71d8bd885e45"}
Oct 30 22:24:49.486491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.486463    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:49.622723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.622693    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-xks-ca\") pod \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") "
Oct 30 22:24:49.622938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.622749    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e359e3-5dd3-429f-9d5a-1d75e5d23473-kube-api-access-pqh8x\" (UniqueName: \"kubernetes.io/projected/09e359e3-5dd3-429f-9d5a-1d75e5d23473-kube-api-access-pqh8x\") pod \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") "
Oct 30 22:24:49.622938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.622780    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-openshift-ca\") pod \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\" (UID: \"09e359e3-5dd3-429f-9d5a-1d75e5d23473\") "
Oct 30 22:24:49.623500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.623468    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-openshift-ca" pod "09e359e3-5dd3-429f-9d5a-1d75e5d23473" (UID: "09e359e3-5dd3-429f-9d5a-1d75e5d23473"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:49.623836 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.623814    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:49.625545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.625513    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-xks-ca" pod "09e359e3-5dd3-429f-9d5a-1d75e5d23473" (UID: "09e359e3-5dd3-429f-9d5a-1d75e5d23473"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:49.625862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.625836    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e359e3-5dd3-429f-9d5a-1d75e5d23473-kube-api-access-pqh8x" pod "09e359e3-5dd3-429f-9d5a-1d75e5d23473" (UID: "09e359e3-5dd3-429f-9d5a-1d75e5d23473"). InnerVolumeSpecName "kube-api-access-pqh8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:49.708941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.708908    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:24:49.722528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.722500    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:24:49.725430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.725282    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e359e3-5dd3-429f-9d5a-1d75e5d23473-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:49.725430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:49.725308    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqh8x\" (UniqueName: \"kubernetes.io/projected/09e359e3-5dd3-429f-9d5a-1d75e5d23473-kube-api-access-pqh8x\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.161065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.161036    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:24:50.177515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.177484    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:50.272670 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.272617    5031 generic.go:356] "Generic (PLEG): container finished" podID="591bc892-852c-416c-9df2-e61a8d9a2015" containerID="d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f" exitCode=0
Oct 30 22:24:50.272902 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.272825    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" event={"ID":"591bc892-852c-416c-9df2-e61a8d9a2015","Type":"ContainerDied","Data":"d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f"}
Oct 30 22:24:50.272902 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.272883    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq" event={"ID":"591bc892-852c-416c-9df2-e61a8d9a2015","Type":"ContainerDied","Data":"eb4a2465115eab1ceb724c045c500772182120b17c8af19a2f272fd1be1312aa"}
Oct 30 22:24:50.273055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.272912    5031 scope.go:122] "RemoveContainer" containerID="d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f"
Oct 30 22:24:50.273139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.273122    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"
Oct 30 22:24:50.276303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.276280    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"
Oct 30 22:24:50.276457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.276271    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt" event={"ID":"09e359e3-5dd3-429f-9d5a-1d75e5d23473","Type":"ContainerDied","Data":"40602462dd421e7fab5e6e6e40ef26b0ef57e5c0de740add2edbae28ee5d99a1"}
Oct 30 22:24:50.276457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.276451    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40602462dd421e7fab5e6e6e40ef26b0ef57e5c0de740add2edbae28ee5d99a1"
Oct 30 22:24:50.287833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.287813    5031 scope.go:122] "RemoveContainer" containerID="5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490"
Oct 30 22:24:50.301111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.301089    5031 scope.go:122] "RemoveContainer" containerID="d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f"
Oct 30 22:24:50.301642 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:24:50.301604    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f\": container with ID starting with d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f not found: ID does not exist" containerID="d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f"
Oct 30 22:24:50.301760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.301637    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f"} err="failed to get container status \"d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f\": rpc error: code = NotFound desc = could not find container \"d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f\": container with ID starting with d21c39133a680921a637ff70edf78683def790751fc38d1349571385d094462f not found: ID does not exist"
Oct 30 22:24:50.301760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.301661    5031 scope.go:122] "RemoveContainer" containerID="5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490"
Oct 30 22:24:50.302013 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:24:50.301989    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490\": container with ID starting with 5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490 not found: ID does not exist" containerID="5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490"
Oct 30 22:24:50.302089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.302024    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490"} err="failed to get container status \"5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490\": rpc error: code = NotFound desc = could not find container \"5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490\": container with ID starting with 5c01983558d06c89657d83c63de670fa9776d7b05fc56cc18ff479eb4cd5c490 not found: ID does not exist"
Oct 30 22:24:50.335532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.335497    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-socket\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.335724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.335619    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591bc892-852c-416c-9df2-e61a8d9a2015-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591bc892-852c-416c-9df2-e61a8d9a2015-istiod-ca-cert\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.335802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.335761    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/591bc892-852c-416c-9df2-e61a8d9a2015-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591bc892-852c-416c-9df2-e61a8d9a2015-istio-podinfo\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.335874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.335804    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-istio-token\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-istio-token\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.335874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.335832    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-socket" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:50.335874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.335866    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-data\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.336005 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.335907    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-envoy\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.336068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.336021    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-certs\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.336068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.336059    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-kube-api-access-m64sd\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-kube-api-access-m64sd\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.336171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.336116    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-credential-socket\") pod \"591bc892-852c-416c-9df2-e61a8d9a2015\" (UID: \"591bc892-852c-416c-9df2-e61a8d9a2015\") "
Oct 30 22:24:50.336249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.336202    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591bc892-852c-416c-9df2-e61a8d9a2015-istiod-ca-cert" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:50.336249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.336235    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-data" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:50.336421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.336391    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-certs" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:50.336624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.336599    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-credential-socket" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:50.337138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.337079    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.337138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.337107    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.337138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.337122    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.337138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.337137    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591bc892-852c-416c-9df2-e61a8d9a2015-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.337138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.337150    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.338860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.338838    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-istio-token" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:50.339007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.338984    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/591bc892-852c-416c-9df2-e61a8d9a2015-istio-podinfo" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:50.339292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.339270    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-kube-api-access-m64sd" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "kube-api-access-m64sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:50.339478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.339457    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-envoy" pod "591bc892-852c-416c-9df2-e61a8d9a2015" (UID: "591bc892-852c-416c-9df2-e61a8d9a2015"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:50.439760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.439650    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591bc892-852c-416c-9df2-e61a8d9a2015-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.439760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.439687    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.439760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.439701    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591bc892-852c-416c-9df2-e61a8d9a2015-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.439760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.439715    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m64sd\" (UniqueName: \"kubernetes.io/projected/591bc892-852c-416c-9df2-e61a8d9a2015-kube-api-access-m64sd\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:50.555940 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:24:50.555905    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:24:50.708681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.708545    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"]
Oct 30 22:24:50.714070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.714018    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq"]
Oct 30 22:24:50.898198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:50.898168    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:51.568694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:51.568610    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:51.803944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:51.803879    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.036187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.036128    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.327617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.327501    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.513087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.513055    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"]
Oct 30 22:24:52.561489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.561457    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.563410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.563388    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-px7cr"]
Oct 30 22:24:52.649742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.649702    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" path="/var/lib/kubelet/pods/591bc892-852c-416c-9df2-e61a8d9a2015/volumes"
Oct 30 22:24:52.650271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.650201    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d802ee16-0512-4053-a1e2-d12ab06210f9" path="/var/lib/kubelet/pods/d802ee16-0512-4053-a1e2-d12ab06210f9/volumes"
Oct 30 22:24:52.821673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:52.821633    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.115234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:53.110390    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.873784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:53.873753    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:54.320475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.320376    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:54.347106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347073    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"]
Oct 30 22:24:54.347814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347791    5031 cpu_manager.go:414] "containerMap: removing container" podUID="09e359e3-5dd3-429f-9d5a-1d75e5d23473" containerName="revoke-keys"
Oct 30 22:24:54.347867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347816    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e359e3-5dd3-429f-9d5a-1d75e5d23473" containerName="revoke-keys"
Oct 30 22:24:54.347867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347849    5031 cpu_manager.go:414] "containerMap: removing container" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy"
Oct 30 22:24:54.347867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347854    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy"
Oct 30 22:24:54.347966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347878    5031 cpu_manager.go:414] "containerMap: removing container" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy"
Oct 30 22:24:54.347966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347885    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy"
Oct 30 22:24:54.348035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347985    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e359e3-5dd3-429f-9d5a-1d75e5d23473" containerName="revoke-keys"
Oct 30 22:24:54.348035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.347997    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy"
Oct 30 22:24:54.348035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.348006    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="591bc892-852c-416c-9df2-e61a8d9a2015" containerName="istio-proxy"
Oct 30 22:24:54.353480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.353462    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.408382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.408348    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:54.408651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.408626    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:24:54.429038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.429004    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkd6\" (UniqueName: \"kubernetes.io/projected/7656e4da-091b-4fdd-9b46-cda1181ecedf-kube-api-access-5pkd6\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.429284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.429264    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.429526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.429499    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.459160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.459132    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:54.459425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.459250    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:54.465944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.465916    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"]
Oct 30 22:24:54.530474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.530437    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.530654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.530499    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkd6\" (UniqueName: \"kubernetes.io/projected/7656e4da-091b-4fdd-9b46-cda1181ecedf-kube-api-access-5pkd6\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.530727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.530707    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.531381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.531362    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.533363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.533341    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.568296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.563437    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:54.588909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.588828    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"]
Oct 30 22:24:54.589139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.589112    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy" containerID="cri-o://f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6" gracePeriod=30
Oct 30 22:24:54.598713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.598684    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkd6\" (UniqueName: \"kubernetes.io/projected/7656e4da-091b-4fdd-9b46-cda1181ecedf-kube-api-access-5pkd6\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.672458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.672415    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:54.802157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:54.802110    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:55.188415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.188384    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"]
Oct 30 22:24:55.189030 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:24:55.189007    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7656e4da_091b_4fdd_9b46_cda1181ecedf.slice/crio-6d590f16b71c1e1208699596ccc726dc6fb1ce819680c56970fb34e8c8a538a1 WatchSource:0}: Error finding container 6d590f16b71c1e1208699596ccc726dc6fb1ce819680c56970fb34e8c8a538a1: Status 404 returned error can't find the container with id 6d590f16b71c1e1208699596ccc726dc6fb1ce819680c56970fb34e8c8a538a1
Oct 30 22:24:55.237582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.237555    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"]
Oct 30 22:24:55.243454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.243434    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.302346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.302317    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"]
Oct 30 22:24:55.348613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.348577    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.348771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.348741    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8kjn\" (UniqueName: \"kubernetes.io/projected/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-kube-api-access-d8kjn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.348812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.348784    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.379179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.379152    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz" event={"ID":"7656e4da-091b-4fdd-9b46-cda1181ecedf","Type":"ContainerStarted","Data":"6d590f16b71c1e1208699596ccc726dc6fb1ce819680c56970fb34e8c8a538a1"}
Oct 30 22:24:55.450269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.450172    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8kjn\" (UniqueName: \"kubernetes.io/projected/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-kube-api-access-d8kjn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.450269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.450234    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.450538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.450379    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.450957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.450930    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.452951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.452924    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.523585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.523562    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8kjn\" (UniqueName: \"kubernetes.io/projected/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-kube-api-access-d8kjn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.558276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.558244    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:55.632848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.632798    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"]
Oct 30 22:24:55.638406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.637601    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.671272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.670838    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.671272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.670984    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.671272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.671039    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfj5q\" (UniqueName: \"kubernetes.io/projected/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-kube-api-access-sfj5q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.690990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.690950    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:55.704027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.703926    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"]
Oct 30 22:24:55.772512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.772479    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.772721 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.772581    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.772721 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.772627    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfj5q\" (UniqueName: \"kubernetes.io/projected/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-kube-api-access-sfj5q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.773366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.773341    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.775224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.775189    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.836967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.836934    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfj5q\" (UniqueName: \"kubernetes.io/projected/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-kube-api-access-sfj5q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:55.949257 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:24:55.949199    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ac7763_9f61_4dc5_8e7f_7d24b90bcae8.slice/crio-1db5f6a66cb66e7b002043259d1d6b30ecdecace9fb9eb89379fe8e81aafdcc0 WatchSource:0}: Error finding container 1db5f6a66cb66e7b002043259d1d6b30ecdecace9fb9eb89379fe8e81aafdcc0: Status 404 returned error can't find the container with id 1db5f6a66cb66e7b002043259d1d6b30ecdecace9fb9eb89379fe8e81aafdcc0
Oct 30 22:24:55.950603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.950570    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"]
Oct 30 22:24:55.978467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:55.978441    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:56.391458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:56.391420    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n" event={"ID":"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8","Type":"ContainerStarted","Data":"4e1fac08db124a8b7b0abab8f54d0742c68fe583b2777420534e661811507131"}
Oct 30 22:24:56.391953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:56.391488    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n" event={"ID":"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8","Type":"ContainerStarted","Data":"1db5f6a66cb66e7b002043259d1d6b30ecdecace9fb9eb89379fe8e81aafdcc0"}
Oct 30 22:24:56.393478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:56.393441    5031 generic.go:356] "Generic (PLEG): container finished" podID="7656e4da-091b-4fdd-9b46-cda1181ecedf" containerID="5ed85b42ae3e4809738f376e0f0d963f6b53609246eee4f6bbe8eae1446b2bc4" exitCode=0
Oct 30 22:24:56.393624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:56.393497    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz" event={"ID":"7656e4da-091b-4fdd-9b46-cda1181ecedf","Type":"ContainerDied","Data":"5ed85b42ae3e4809738f376e0f0d963f6b53609246eee4f6bbe8eae1446b2bc4"}
Oct 30 22:24:56.474045 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:24:56.474016    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc8ee34_a224_46e7_9865_cdd4eacf5bfc.slice/crio-b695ade020d557d90c5fcf50c99b33b268a0100cdddc37593cf5ee5d11acd80c WatchSource:0}: Error finding container b695ade020d557d90c5fcf50c99b33b268a0100cdddc37593cf5ee5d11acd80c: Status 404 returned error can't find the container with id b695ade020d557d90c5fcf50c99b33b268a0100cdddc37593cf5ee5d11acd80c
Oct 30 22:24:56.501989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:56.501951    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"]
Oct 30 22:24:56.522306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:56.522258    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n" podStartSLOduration=1.5222440289999999 podStartE2EDuration="1.522244029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:56.521010015 +0000 UTC m=+2108.045152791" watchObservedRunningTime="2026-10-30 22:24:56.522244029 +0000 UTC m=+2108.046386797"
Oct 30 22:24:57.403996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.403900    5031 generic.go:356] "Generic (PLEG): container finished" podID="d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" containerID="4e1fac08db124a8b7b0abab8f54d0742c68fe583b2777420534e661811507131" exitCode=0
Oct 30 22:24:57.404515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.403987    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n" event={"ID":"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8","Type":"ContainerDied","Data":"4e1fac08db124a8b7b0abab8f54d0742c68fe583b2777420534e661811507131"}
Oct 30 22:24:57.405914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.405891    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf" event={"ID":"edc8ee34-a224-46e7-9865-cdd4eacf5bfc","Type":"ContainerStarted","Data":"c59a46dd45cc7f8236f8548902b3f0bafe2ea51c79c0f3b635b27afd971424f6"}
Oct 30 22:24:57.406076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.405929    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf" event={"ID":"edc8ee34-a224-46e7-9865-cdd4eacf5bfc","Type":"ContainerStarted","Data":"b695ade020d557d90c5fcf50c99b33b268a0100cdddc37593cf5ee5d11acd80c"}
Oct 30 22:24:57.650471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.650443    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:57.664909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.664465    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf" podStartSLOduration=2.664446657 podStartE2EDuration="2.664446657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:57.657132166 +0000 UTC m=+2109.181274957" watchObservedRunningTime="2026-10-30 22:24:57.664446657 +0000 UTC m=+2109.188589421"
Oct 30 22:24:57.711312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.711281    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-openshift-ca\") pod \"7656e4da-091b-4fdd-9b46-cda1181ecedf\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") "
Oct 30 22:24:57.711485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.711452    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-xks-ca\") pod \"7656e4da-091b-4fdd-9b46-cda1181ecedf\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") "
Oct 30 22:24:57.711575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.711560    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7656e4da-091b-4fdd-9b46-cda1181ecedf-kube-api-access-5pkd6\" (UniqueName: \"kubernetes.io/projected/7656e4da-091b-4fdd-9b46-cda1181ecedf-kube-api-access-5pkd6\") pod \"7656e4da-091b-4fdd-9b46-cda1181ecedf\" (UID: \"7656e4da-091b-4fdd-9b46-cda1181ecedf\") "
Oct 30 22:24:57.711672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.711647    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-openshift-ca" pod "7656e4da-091b-4fdd-9b46-cda1181ecedf" (UID: "7656e4da-091b-4fdd-9b46-cda1181ecedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:57.712188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.712157    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.713791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.713770    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-xks-ca" pod "7656e4da-091b-4fdd-9b46-cda1181ecedf" (UID: "7656e4da-091b-4fdd-9b46-cda1181ecedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:57.713966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.713946    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7656e4da-091b-4fdd-9b46-cda1181ecedf-kube-api-access-5pkd6" pod "7656e4da-091b-4fdd-9b46-cda1181ecedf" (UID: "7656e4da-091b-4fdd-9b46-cda1181ecedf"). InnerVolumeSpecName "kube-api-access-5pkd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:57.813699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.813664    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7656e4da-091b-4fdd-9b46-cda1181ecedf-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:57.813699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:57.813696    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkd6\" (UniqueName: \"kubernetes.io/projected/7656e4da-091b-4fdd-9b46-cda1181ecedf-kube-api-access-5pkd6\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.045298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.044559    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:58.418283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.418247    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"
Oct 30 22:24:58.418283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.418278    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz" event={"ID":"7656e4da-091b-4fdd-9b46-cda1181ecedf","Type":"ContainerDied","Data":"6d590f16b71c1e1208699596ccc726dc6fb1ce819680c56970fb34e8c8a538a1"}
Oct 30 22:24:58.418871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.418328    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d590f16b71c1e1208699596ccc726dc6fb1ce819680c56970fb34e8c8a538a1"
Oct 30 22:24:58.421251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.421186    5031 generic.go:356] "Generic (PLEG): container finished" podID="edc8ee34-a224-46e7-9865-cdd4eacf5bfc" containerID="c59a46dd45cc7f8236f8548902b3f0bafe2ea51c79c0f3b635b27afd971424f6" exitCode=0
Oct 30 22:24:58.421413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.421392    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf" event={"ID":"edc8ee34-a224-46e7-9865-cdd4eacf5bfc","Type":"ContainerDied","Data":"c59a46dd45cc7f8236f8548902b3f0bafe2ea51c79c0f3b635b27afd971424f6"}
Oct 30 22:24:58.541760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.540816    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:58.677817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.677737    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:58.754891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.754855    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"]
Oct 30 22:24:58.780636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.780606    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"]
Oct 30 22:24:58.787382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.787338    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:58.789806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.789774    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"]
Oct 30 22:24:58.831863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.831832    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-openshift-ca\") pod \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") "
Oct 30 22:24:58.832041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.832012    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-kube-api-access-d8kjn\" (UniqueName: \"kubernetes.io/projected/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-kube-api-access-d8kjn\") pod \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") "
Oct 30 22:24:58.832273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.832249    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-openshift-ca" pod "d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" (UID: "d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:58.832387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.832251    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-xks-ca\") pod \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\" (UID: \"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8\") "
Oct 30 22:24:58.833182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.833150    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.834715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.834690    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-kube-api-access-d8kjn" pod "d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" (UID: "d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8"). InnerVolumeSpecName "kube-api-access-d8kjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:58.835389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.835366    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-xks-ca" pod "d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" (UID: "d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:58.934526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.934436    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.934526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:58.934466    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8kjn\" (UniqueName: \"kubernetes.io/projected/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8-kube-api-access-d8kjn\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.092312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.092266    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:59.433281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.433250    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"
Oct 30 22:24:59.433281 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.433250    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n" event={"ID":"d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8","Type":"ContainerDied","Data":"1db5f6a66cb66e7b002043259d1d6b30ecdecace9fb9eb89379fe8e81aafdcc0"}
Oct 30 22:24:59.433774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.433311    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db5f6a66cb66e7b002043259d1d6b30ecdecace9fb9eb89379fe8e81aafdcc0"
Oct 30 22:24:59.473850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.473816    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:59.606136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.606108    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:24:59.744729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.744684    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-kube-api-access-sfj5q\" (UniqueName: \"kubernetes.io/projected/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-kube-api-access-sfj5q\") pod \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") "
Oct 30 22:24:59.744988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.744764    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-openshift-ca\") pod \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") "
Oct 30 22:24:59.744988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.744917    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-xks-ca\") pod \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\" (UID: \"edc8ee34-a224-46e7-9865-cdd4eacf5bfc\") "
Oct 30 22:24:59.746812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.746784    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-openshift-ca" pod "edc8ee34-a224-46e7-9865-cdd4eacf5bfc" (UID: "edc8ee34-a224-46e7-9865-cdd4eacf5bfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:59.752705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.752670    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-xks-ca" pod "edc8ee34-a224-46e7-9865-cdd4eacf5bfc" (UID: "edc8ee34-a224-46e7-9865-cdd4eacf5bfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:59.753191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.753166    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-kube-api-access-sfj5q" pod "edc8ee34-a224-46e7-9865-cdd4eacf5bfc" (UID: "edc8ee34-a224-46e7-9865-cdd4eacf5bfc"). InnerVolumeSpecName "kube-api-access-sfj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:59.849999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.849938    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfj5q\" (UniqueName: \"kubernetes.io/projected/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-kube-api-access-sfj5q\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.877515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.849979    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.877515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.850251    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc8ee34-a224-46e7-9865-cdd4eacf5bfc-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.877515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:24:59.852285    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:00.065189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.065155    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:25:00.157029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.156997    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-socket\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157092    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-token\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-token\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157160    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-certs\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157186    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-kube-api-access-kd6vp\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-kube-api-access-kd6vp\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157259    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-data\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157311    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-credential-socket\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157310    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-socket" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:00.157530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157389    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-podinfo\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157467    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895bc537-7f1c-4950-bfa0-b89ae9531f95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895bc537-7f1c-4950-bfa0-b89ae9531f95-istiod-ca-cert\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157506    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-envoy\") pod \"895bc537-7f1c-4950-bfa0-b89ae9531f95\" (UID: \"895bc537-7f1c-4950-bfa0-b89ae9531f95\") "
Oct 30 22:25:00.157694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157596    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-data" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:00.157928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.157900    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895bc537-7f1c-4950-bfa0-b89ae9531f95-istiod-ca-cert" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:00.158048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.158034    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-credential-socket" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:00.158331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.158119    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-certs" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:00.158463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.158439    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.158542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.158471    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.158542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.158487    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.158542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.158500    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895bc537-7f1c-4950-bfa0-b89ae9531f95-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.158542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.158517    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.159806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.159770    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-token" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:00.159897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.159801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-podinfo" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:00.159897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.159816    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-kube-api-access-kd6vp" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "kube-api-access-kd6vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:00.159965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.159941    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-envoy" pod "895bc537-7f1c-4950-bfa0-b89ae9531f95" (UID: "895bc537-7f1c-4950-bfa0-b89ae9531f95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:00.260635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.260548    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.260635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.260590    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.260635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.260606    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd6vp\" (UniqueName: \"kubernetes.io/projected/895bc537-7f1c-4950-bfa0-b89ae9531f95-kube-api-access-kd6vp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.260635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.260621    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895bc537-7f1c-4950-bfa0-b89ae9531f95-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:00.446333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.446289    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf" event={"ID":"edc8ee34-a224-46e7-9865-cdd4eacf5bfc","Type":"ContainerDied","Data":"b695ade020d557d90c5fcf50c99b33b268a0100cdddc37593cf5ee5d11acd80c"}
Oct 30 22:25:00.446822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.446303    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"
Oct 30 22:25:00.446905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.446838    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b695ade020d557d90c5fcf50c99b33b268a0100cdddc37593cf5ee5d11acd80c"
Oct 30 22:25:00.448889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.448858    5031 generic.go:356] "Generic (PLEG): container finished" podID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerID="f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6" exitCode=0
Oct 30 22:25:00.449057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.448925    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"
Oct 30 22:25:00.449057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.449045    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" event={"ID":"895bc537-7f1c-4950-bfa0-b89ae9531f95","Type":"ContainerDied","Data":"f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6"}
Oct 30 22:25:00.449160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.449089    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b" event={"ID":"895bc537-7f1c-4950-bfa0-b89ae9531f95","Type":"ContainerDied","Data":"cf47ac9911ff499c3ff4550a22f98ea4e474b2c00d981cc8f90c5c6cf1bfc766"}
Oct 30 22:25:00.449160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.449120    5031 scope.go:122] "RemoveContainer" containerID="f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6"
Oct 30 22:25:00.463365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.463342    5031 scope.go:122] "RemoveContainer" containerID="c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2"
Oct 30 22:25:00.475185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.475152    5031 scope.go:122] "RemoveContainer" containerID="f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6"
Oct 30 22:25:00.475495 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:00.475474    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6\": container with ID starting with f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6 not found: ID does not exist" containerID="f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6"
Oct 30 22:25:00.475576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.475516    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6"} err="failed to get container status \"f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6\": rpc error: code = NotFound desc = could not find container \"f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6\": container with ID starting with f6176c739ce7e9e248f5f5caacbddd98664af17a16650e6ef95669e3bf036ac6 not found: ID does not exist"
Oct 30 22:25:00.475576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.475536    5031 scope.go:122] "RemoveContainer" containerID="c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2"
Oct 30 22:25:00.475833 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:00.475814    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2\": container with ID starting with c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2 not found: ID does not exist" containerID="c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2"
Oct 30 22:25:00.475889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.475841    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2"} err="failed to get container status \"c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2\": rpc error: code = NotFound desc = could not find container \"c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2\": container with ID starting with c43f8328a9f867926b652ef6d85b01e6f48e00b0f3ad4f4d520a59862917ffa2 not found: ID does not exist"
Oct 30 22:25:00.676546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.676517    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"]
Oct 30 22:25:00.680554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:00.680532    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b"]
Oct 30 22:25:02.425670 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:02.425635    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:02.671269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:02.667391    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" path="/var/lib/kubelet/pods/895bc537-7f1c-4950-bfa0-b89ae9531f95/volumes"
Oct 30 22:25:02.696652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:02.696129    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:04.162076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:04.162043    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:04.741250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:04.741204    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"]
Oct 30 22:25:05.220584 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:05.220552    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:05.460171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:05.460138    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:07.751612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:07.751578    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:07.985170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:07.985140    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:09.006420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.006386    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"]
Oct 30 22:25:09.007154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007128    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7656e4da-091b-4fdd-9b46-cda1181ecedf" containerName="revoke-keys"
Oct 30 22:25:09.007154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007155    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7656e4da-091b-4fdd-9b46-cda1181ecedf" containerName="revoke-keys"
Oct 30 22:25:09.007316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007199    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" containerName="revoke-keys"
Oct 30 22:25:09.007316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007226    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" containerName="revoke-keys"
Oct 30 22:25:09.007316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007245    5031 cpu_manager.go:414] "containerMap: removing container" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy"
Oct 30 22:25:09.007316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007253    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy"
Oct 30 22:25:09.007316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007288    5031 cpu_manager.go:414] "containerMap: removing container" podUID="edc8ee34-a224-46e7-9865-cdd4eacf5bfc" containerName="revoke-keys"
Oct 30 22:25:09.007316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007295    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc8ee34-a224-46e7-9865-cdd4eacf5bfc" containerName="revoke-keys"
Oct 30 22:25:09.007508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007473    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7656e4da-091b-4fdd-9b46-cda1181ecedf" containerName="revoke-keys"
Oct 30 22:25:09.007508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007494    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" containerName="revoke-keys"
Oct 30 22:25:09.007508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007504    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy"
Oct 30 22:25:09.007607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007515    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy"
Oct 30 22:25:09.007607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.007528    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc8ee34-a224-46e7-9865-cdd4eacf5bfc" containerName="revoke-keys"
Oct 30 22:25:09.016592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.016562    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.089785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.089666    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.090241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.090193    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.090674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.090522    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwpx\" (UniqueName: \"kubernetes.io/projected/eb469053-f5e2-4e90-9f98-be803e987d25-kube-api-access-6cwpx\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.098244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.096202    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:09.098244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.096781    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:09.122261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.122226    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"]
Oct 30 22:25:09.123564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.123517    5031 cpu_manager.go:414] "containerMap: removing container" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy"
Oct 30 22:25:09.123775 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.123761    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895bc537-7f1c-4950-bfa0-b89ae9531f95" containerName="istio-proxy"
Oct 30 22:25:09.134073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.133629    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.134073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.133918    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"]
Oct 30 22:25:09.144320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.144291    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"]
Oct 30 22:25:09.171482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.159177    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:25:09.171482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.169498    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:09.194608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.194551    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.195036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.195011    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwpx\" (UniqueName: \"kubernetes.io/projected/eb469053-f5e2-4e90-9f98-be803e987d25-kube-api-access-6cwpx\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.195431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.195382    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.196602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.196576    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.196794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.196775    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.196922 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.196906    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrnl\" (UniqueName: \"kubernetes.io/projected/414303a8-eacb-4472-8115-f8be154c96d1-kube-api-access-fvrnl\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.200604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.200579    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.230965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.230874    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.262044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.261976    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwpx\" (UniqueName: \"kubernetes.io/projected/eb469053-f5e2-4e90-9f98-be803e987d25-kube-api-access-6cwpx\") pod \"maas-api-revoke-sub-ffd38b61-vzwnn\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.298758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.298728    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrnl\" (UniqueName: \"kubernetes.io/projected/414303a8-eacb-4472-8115-f8be154c96d1-kube-api-access-fvrnl\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.298911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.298812    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.298969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.298931    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.299600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.299574    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.301428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.301404    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.365787 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.365752    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrnl\" (UniqueName: \"kubernetes.io/projected/414303a8-eacb-4472-8115-f8be154c96d1-kube-api-access-fvrnl\") pod \"maas-api-revoke-sub-00e78729-fcpwk\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.366004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.365989    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:09.532038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.531893    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:09.741603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.741560    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" podUID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" containerName="payload-processing" containerID="cri-o://4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9" gracePeriod=25
Oct 30 22:25:09.765688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.765655    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"]
Oct 30 22:25:09.765942 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:25:09.765918    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb469053_f5e2_4e90_9f98_be803e987d25.slice/crio-0187db5569d23bbcfd9664a353c28298623c420bc4a8fa2a8cec4ba9ef707d4d WatchSource:0}: Error finding container 0187db5569d23bbcfd9664a353c28298623c420bc4a8fa2a8cec4ba9ef707d4d: Status 404 returned error can't find the container with id 0187db5569d23bbcfd9664a353c28298623c420bc4a8fa2a8cec4ba9ef707d4d
Oct 30 22:25:09.937162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:09.937138    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"]
Oct 30 22:25:09.941720 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:25:09.941686    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414303a8_eacb_4472_8115_f8be154c96d1.slice/crio-873c18cd62e4a7e79ee1785443efc753fe5379a08b8736d6f33d39cdedcf64f7 WatchSource:0}: Error finding container 873c18cd62e4a7e79ee1785443efc753fe5379a08b8736d6f33d39cdedcf64f7: Status 404 returned error can't find the container with id 873c18cd62e4a7e79ee1785443efc753fe5379a08b8736d6f33d39cdedcf64f7
Oct 30 22:25:10.074334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.074245    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.379233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.376684    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.602729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.602695    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn" event={"ID":"eb469053-f5e2-4e90-9f98-be803e987d25","Type":"ContainerStarted","Data":"98d479e2e87be2e1230aa34eaca6da18a8725ea4f36a02ee21491ee42fa388ad"}
Oct 30 22:25:10.602922 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.602760    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn" event={"ID":"eb469053-f5e2-4e90-9f98-be803e987d25","Type":"ContainerStarted","Data":"0187db5569d23bbcfd9664a353c28298623c420bc4a8fa2a8cec4ba9ef707d4d"}
Oct 30 22:25:10.604556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.604527    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk" event={"ID":"414303a8-eacb-4472-8115-f8be154c96d1","Type":"ContainerStarted","Data":"612717761cc6c302b540b6633000b500b6d1fc7a7e51b748b2058edc38b3ca91"}
Oct 30 22:25:10.604668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.604576    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk" event={"ID":"414303a8-eacb-4472-8115-f8be154c96d1","Type":"ContainerStarted","Data":"873c18cd62e4a7e79ee1785443efc753fe5379a08b8736d6f33d39cdedcf64f7"}
Oct 30 22:25:10.691300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.691267    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.728143 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.728083    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn" podStartSLOduration=2.728068811 podStartE2EDuration="2.728068811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:10.727023904 +0000 UTC m=+2122.251166669" watchObservedRunningTime="2026-10-30 22:25:10.728068811 +0000 UTC m=+2122.252211574"
Oct 30 22:25:10.850624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:10.850560    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk" podStartSLOduration=2.85054396 podStartE2EDuration="2.85054396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:10.847575691 +0000 UTC m=+2122.371718457" watchObservedRunningTime="2026-10-30 22:25:10.85054396 +0000 UTC m=+2122.374686803"
Oct 30 22:25:11.621255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:11.618192    5031 generic.go:356] "Generic (PLEG): container finished" podID="eb469053-f5e2-4e90-9f98-be803e987d25" containerID="98d479e2e87be2e1230aa34eaca6da18a8725ea4f36a02ee21491ee42fa388ad" exitCode=0
Oct 30 22:25:11.621255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:11.618254    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn" event={"ID":"eb469053-f5e2-4e90-9f98-be803e987d25","Type":"ContainerDied","Data":"98d479e2e87be2e1230aa34eaca6da18a8725ea4f36a02ee21491ee42fa388ad"}
Oct 30 22:25:11.621255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:11.620364    5031 generic.go:356] "Generic (PLEG): container finished" podID="414303a8-eacb-4472-8115-f8be154c96d1" containerID="612717761cc6c302b540b6633000b500b6d1fc7a7e51b748b2058edc38b3ca91" exitCode=0
Oct 30 22:25:11.621255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:11.620443    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk" event={"ID":"414303a8-eacb-4472-8115-f8be154c96d1","Type":"ContainerDied","Data":"612717761cc6c302b540b6633000b500b6d1fc7a7e51b748b2058edc38b3ca91"}
Oct 30 22:25:12.840313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.840285    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:12.845002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.844979    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:12.892956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.892866    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-xks-ca\") pod \"eb469053-f5e2-4e90-9f98-be803e987d25\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") "
Oct 30 22:25:12.892956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.892931    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-xks-ca\") pod \"414303a8-eacb-4472-8115-f8be154c96d1\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") "
Oct 30 22:25:12.893300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.893153    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414303a8-eacb-4472-8115-f8be154c96d1-kube-api-access-fvrnl\" (UniqueName: \"kubernetes.io/projected/414303a8-eacb-4472-8115-f8be154c96d1-kube-api-access-fvrnl\") pod \"414303a8-eacb-4472-8115-f8be154c96d1\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") "
Oct 30 22:25:12.893300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.893249    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb469053-f5e2-4e90-9f98-be803e987d25-kube-api-access-6cwpx\" (UniqueName: \"kubernetes.io/projected/eb469053-f5e2-4e90-9f98-be803e987d25-kube-api-access-6cwpx\") pod \"eb469053-f5e2-4e90-9f98-be803e987d25\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") "
Oct 30 22:25:12.893300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.893282    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-openshift-ca\") pod \"414303a8-eacb-4472-8115-f8be154c96d1\" (UID: \"414303a8-eacb-4472-8115-f8be154c96d1\") "
Oct 30 22:25:12.893500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.893340    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-openshift-ca\") pod \"eb469053-f5e2-4e90-9f98-be803e987d25\" (UID: \"eb469053-f5e2-4e90-9f98-be803e987d25\") "
Oct 30 22:25:12.893776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.893746    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-openshift-ca" pod "414303a8-eacb-4472-8115-f8be154c96d1" (UID: "414303a8-eacb-4472-8115-f8be154c96d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:12.893858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.893783    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-openshift-ca" pod "eb469053-f5e2-4e90-9f98-be803e987d25" (UID: "eb469053-f5e2-4e90-9f98-be803e987d25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:12.895940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.895876    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.895940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.895904    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.895940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.895901    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-xks-ca" pod "eb469053-f5e2-4e90-9f98-be803e987d25" (UID: "eb469053-f5e2-4e90-9f98-be803e987d25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:12.896567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.896128    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414303a8-eacb-4472-8115-f8be154c96d1-kube-api-access-fvrnl" pod "414303a8-eacb-4472-8115-f8be154c96d1" (UID: "414303a8-eacb-4472-8115-f8be154c96d1"). InnerVolumeSpecName "kube-api-access-fvrnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:12.896567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.896168    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-xks-ca" pod "414303a8-eacb-4472-8115-f8be154c96d1" (UID: "414303a8-eacb-4472-8115-f8be154c96d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:12.896567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.896293    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb469053-f5e2-4e90-9f98-be803e987d25-kube-api-access-6cwpx" pod "eb469053-f5e2-4e90-9f98-be803e987d25" (UID: "eb469053-f5e2-4e90-9f98-be803e987d25"). InnerVolumeSpecName "kube-api-access-6cwpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:12.997926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.997880    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb469053-f5e2-4e90-9f98-be803e987d25-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.997926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.997924    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414303a8-eacb-4472-8115-f8be154c96d1-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.998202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.997941    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvrnl\" (UniqueName: \"kubernetes.io/projected/414303a8-eacb-4472-8115-f8be154c96d1-kube-api-access-fvrnl\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.998202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:12.997956    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cwpx\" (UniqueName: \"kubernetes.io/projected/eb469053-f5e2-4e90-9f98-be803e987d25-kube-api-access-6cwpx\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:13.651372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.651343    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"
Oct 30 22:25:13.651568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.651372    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn" event={"ID":"eb469053-f5e2-4e90-9f98-be803e987d25","Type":"ContainerDied","Data":"0187db5569d23bbcfd9664a353c28298623c420bc4a8fa2a8cec4ba9ef707d4d"}
Oct 30 22:25:13.651568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.651425    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0187db5569d23bbcfd9664a353c28298623c420bc4a8fa2a8cec4ba9ef707d4d"
Oct 30 22:25:13.653056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.653034    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"
Oct 30 22:25:13.653056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.653049    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk" event={"ID":"414303a8-eacb-4472-8115-f8be154c96d1","Type":"ContainerDied","Data":"873c18cd62e4a7e79ee1785443efc753fe5379a08b8736d6f33d39cdedcf64f7"}
Oct 30 22:25:13.653268 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.653088    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="873c18cd62e4a7e79ee1785443efc753fe5379a08b8736d6f33d39cdedcf64f7"
Oct 30 22:25:13.821613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.821572    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" containerID="cri-o://fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81" gracePeriod=45
Oct 30 22:25:13.821814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.821572    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" containerID="cri-o://55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174" gracePeriod=45
Oct 30 22:25:13.823562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.823531    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main" containerID="cri-o://0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2" gracePeriod=45
Oct 30 22:25:13.848722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.848687    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 22:25:13.853690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:13.853662    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": dial tcp 10.132.0.117:8000: connect: connection refused"
Oct 30 22:25:14.189125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.189052    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:25:14.191953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.191931    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:25:14.196366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.196347    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:25:14.327967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.327932    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b27147da-5731-4444-92d8-58f799907b2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b27147da-5731-4444-92d8-58f799907b2b-tls-certs\") pod \"b27147da-5731-4444-92d8-58f799907b2b\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") "
Oct 30 22:25:14.328191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328005    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-home\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-home\") pod \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") "
Oct 30 22:25:14.328191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328073    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27147da-5731-4444-92d8-58f799907b2b-kube-api-access-cqghs\" (UniqueName: \"kubernetes.io/projected/b27147da-5731-4444-92d8-58f799907b2b-kube-api-access-cqghs\") pod \"b27147da-5731-4444-92d8-58f799907b2b\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") "
Oct 30 22:25:14.328191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328105    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7a540c-99bf-420d-a7eb-ef1664d01bce-kube-api-access-2kb7q\" (UniqueName: \"kubernetes.io/projected/ff7a540c-99bf-420d-a7eb-ef1664d01bce-kube-api-access-2kb7q\") pod \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") "
Oct 30 22:25:14.328405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328203    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-tmp-dir\") pod \"1241795d-784e-4ab5-8b92-66d23a747f3e\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") "
Oct 30 22:25:14.328405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328259    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-home\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-home\") pod \"b27147da-5731-4444-92d8-58f799907b2b\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") "
Oct 30 22:25:14.328405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328291    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tmp-dir\") pod \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") "
Oct 30 22:25:14.328405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328325    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-dshm\") pod \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") "
Oct 30 22:25:14.328405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328381    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-dshm\") pod \"1241795d-784e-4ab5-8b92-66d23a747f3e\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") "
Oct 30 22:25:14.328405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328387    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-home" pod "ff7a540c-99bf-420d-a7eb-ef1664d01bce" (UID: "ff7a540c-99bf-420d-a7eb-ef1664d01bce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328412    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-tmp-dir\") pod \"b27147da-5731-4444-92d8-58f799907b2b\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") "
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328440    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-model-cache\") pod \"b27147da-5731-4444-92d8-58f799907b2b\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") "
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328472    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1241795d-784e-4ab5-8b92-66d23a747f3e-kube-api-access-w5t9n\" (UniqueName: \"kubernetes.io/projected/1241795d-784e-4ab5-8b92-66d23a747f3e-kube-api-access-w5t9n\") pod \"1241795d-784e-4ab5-8b92-66d23a747f3e\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") "
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328515    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-model-cache\") pod \"1241795d-784e-4ab5-8b92-66d23a747f3e\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") "
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328573    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-dshm\") pod \"b27147da-5731-4444-92d8-58f799907b2b\" (UID: \"b27147da-5731-4444-92d8-58f799907b2b\") "
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328610    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-home" pod "b27147da-5731-4444-92d8-58f799907b2b" (UID: "b27147da-5731-4444-92d8-58f799907b2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328603    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-home\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-home\") pod \"1241795d-784e-4ab5-8b92-66d23a747f3e\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") "
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328670    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-model-cache\") pod \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") "
Oct 30 22:25:14.328729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328709    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-tmp-dir" pod "b27147da-5731-4444-92d8-58f799907b2b" (UID: "b27147da-5731-4444-92d8-58f799907b2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.329132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328735    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1241795d-784e-4ab5-8b92-66d23a747f3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1241795d-784e-4ab5-8b92-66d23a747f3e-tls-certs\") pod \"1241795d-784e-4ab5-8b92-66d23a747f3e\" (UID: \"1241795d-784e-4ab5-8b92-66d23a747f3e\") "
Oct 30 22:25:14.329132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328759    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tls-certs\") pod \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\" (UID: \"ff7a540c-99bf-420d-a7eb-ef1664d01bce\") "
Oct 30 22:25:14.329132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.328993    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tmp-dir" pod "ff7a540c-99bf-420d-a7eb-ef1664d01bce" (UID: "ff7a540c-99bf-420d-a7eb-ef1664d01bce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.329310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329287    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-tmp-dir" pod "1241795d-784e-4ab5-8b92-66d23a747f3e" (UID: "1241795d-784e-4ab5-8b92-66d23a747f3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.329583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329526    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-home" pod "1241795d-784e-4ab5-8b92-66d23a747f3e" (UID: "1241795d-784e-4ab5-8b92-66d23a747f3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.329790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329672    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.329790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329694    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.329790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329698    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-model-cache" pod "ff7a540c-99bf-420d-a7eb-ef1664d01bce" (UID: "ff7a540c-99bf-420d-a7eb-ef1664d01bce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.329790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329706    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.329790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329735    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.329790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.329751    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.331397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.331371    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27147da-5731-4444-92d8-58f799907b2b-tls-certs" pod "b27147da-5731-4444-92d8-58f799907b2b" (UID: "b27147da-5731-4444-92d8-58f799907b2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:14.331760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.331742    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-model-cache" pod "b27147da-5731-4444-92d8-58f799907b2b" (UID: "b27147da-5731-4444-92d8-58f799907b2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.332125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.332098    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-model-cache" pod "1241795d-784e-4ab5-8b92-66d23a747f3e" (UID: "1241795d-784e-4ab5-8b92-66d23a747f3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.332223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.332177    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-dshm" pod "ff7a540c-99bf-420d-a7eb-ef1664d01bce" (UID: "ff7a540c-99bf-420d-a7eb-ef1664d01bce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.333290 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.333255    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-dshm" pod "b27147da-5731-4444-92d8-58f799907b2b" (UID: "b27147da-5731-4444-92d8-58f799907b2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.333410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.333391    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7a540c-99bf-420d-a7eb-ef1664d01bce-kube-api-access-2kb7q" pod "ff7a540c-99bf-420d-a7eb-ef1664d01bce" (UID: "ff7a540c-99bf-420d-a7eb-ef1664d01bce"). InnerVolumeSpecName "kube-api-access-2kb7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:14.334593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.334483    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1241795d-784e-4ab5-8b92-66d23a747f3e-kube-api-access-w5t9n" pod "1241795d-784e-4ab5-8b92-66d23a747f3e" (UID: "1241795d-784e-4ab5-8b92-66d23a747f3e"). InnerVolumeSpecName "kube-api-access-w5t9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:14.334593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.334568    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-dshm" pod "1241795d-784e-4ab5-8b92-66d23a747f3e" (UID: "1241795d-784e-4ab5-8b92-66d23a747f3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.334744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.334687    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1241795d-784e-4ab5-8b92-66d23a747f3e-tls-certs" pod "1241795d-784e-4ab5-8b92-66d23a747f3e" (UID: "1241795d-784e-4ab5-8b92-66d23a747f3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:14.335065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.335046    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27147da-5731-4444-92d8-58f799907b2b-kube-api-access-cqghs" pod "b27147da-5731-4444-92d8-58f799907b2b" (UID: "b27147da-5731-4444-92d8-58f799907b2b"). InnerVolumeSpecName "kube-api-access-cqghs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:14.335334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.335312    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tls-certs" pod "ff7a540c-99bf-420d-a7eb-ef1664d01bce" (UID: "ff7a540c-99bf-420d-a7eb-ef1664d01bce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432424    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432455    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432465    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1241795d-784e-4ab5-8b92-66d23a747f3e-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432474    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7a540c-99bf-420d-a7eb-ef1664d01bce-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432482    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27147da-5731-4444-92d8-58f799907b2b-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432492    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqghs\" (UniqueName: \"kubernetes.io/projected/b27147da-5731-4444-92d8-58f799907b2b-kube-api-access-cqghs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432504    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kb7q\" (UniqueName: \"kubernetes.io/projected/ff7a540c-99bf-420d-a7eb-ef1664d01bce-kube-api-access-2kb7q\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432515    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7a540c-99bf-420d-a7eb-ef1664d01bce-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432527    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432538    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432550    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5t9n\" (UniqueName: \"kubernetes.io/projected/1241795d-784e-4ab5-8b92-66d23a747f3e-kube-api-access-w5t9n\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432565    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1241795d-784e-4ab5-8b92-66d23a747f3e-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.432599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.432573    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27147da-5731-4444-92d8-58f799907b2b-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.579576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.579483    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:14.665260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.665225    5031 generic.go:356] "Generic (PLEG): container finished" podID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerID="0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2" exitCode=0
Oct 30 22:25:14.665368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.665271    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"
Oct 30 22:25:14.665368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.665324    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" event={"ID":"ff7a540c-99bf-420d-a7eb-ef1664d01bce","Type":"ContainerDied","Data":"0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2"}
Oct 30 22:25:14.665463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.665369    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz" event={"ID":"ff7a540c-99bf-420d-a7eb-ef1664d01bce","Type":"ContainerDied","Data":"1051d13b79f78819295503ed7446287c476775701e1dbc2b023abb8e3678f6a0"}
Oct 30 22:25:14.665463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.665389    5031 scope.go:122] "RemoveContainer" containerID="0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2"
Oct 30 22:25:14.668533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.667648    5031 generic.go:356] "Generic (PLEG): container finished" podID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerID="55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174" exitCode=0
Oct 30 22:25:14.668533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.667851    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" event={"ID":"1241795d-784e-4ab5-8b92-66d23a747f3e","Type":"ContainerDied","Data":"55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174"}
Oct 30 22:25:14.668533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.667882    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w" event={"ID":"1241795d-784e-4ab5-8b92-66d23a747f3e","Type":"ContainerDied","Data":"08c77629bdc74242bd661293080e183dd482f3a2c8d8d0b6284e127110520027"}
Oct 30 22:25:14.668533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.667854    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"
Oct 30 22:25:14.672732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.670161    5031 generic.go:356] "Generic (PLEG): container finished" podID="b27147da-5731-4444-92d8-58f799907b2b" containerID="fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81" exitCode=0
Oct 30 22:25:14.672732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.670254    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" event={"ID":"b27147da-5731-4444-92d8-58f799907b2b","Type":"ContainerDied","Data":"fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81"}
Oct 30 22:25:14.672732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.670281    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv" event={"ID":"b27147da-5731-4444-92d8-58f799907b2b","Type":"ContainerDied","Data":"fb0742930c9d8d6f2c030ade4b9313d24148e0041acda8a50e32142dabb175ee"}
Oct 30 22:25:14.672732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.670344    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"
Oct 30 22:25:14.681814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.681793    5031 scope.go:122] "RemoveContainer" containerID="0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2"
Oct 30 22:25:14.682164 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:14.682143    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2\": container with ID starting with 0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2 not found: ID does not exist" containerID="0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2"
Oct 30 22:25:14.682247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.682169    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2"} err="failed to get container status \"0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2\": rpc error: code = NotFound desc = could not find container \"0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2\": container with ID starting with 0537d4a0436a2a04699d0df391f23e95c2cb88d7e4891bb6a3f445f0112577e2 not found: ID does not exist"
Oct 30 22:25:14.682247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.682187    5031 scope.go:122] "RemoveContainer" containerID="55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174"
Oct 30 22:25:14.693052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.693029    5031 scope.go:122] "RemoveContainer" containerID="55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174"
Oct 30 22:25:14.693403 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:14.693377    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174\": container with ID starting with 55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174 not found: ID does not exist" containerID="55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174"
Oct 30 22:25:14.693508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.693410    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174"} err="failed to get container status \"55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174\": rpc error: code = NotFound desc = could not find container \"55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174\": container with ID starting with 55a46ac3a3e6d339b9fc87d48bb05df4057187a9d6412d7f4dd19dad9414b174 not found: ID does not exist"
Oct 30 22:25:14.693508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.693428    5031 scope.go:122] "RemoveContainer" containerID="fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81"
Oct 30 22:25:14.704043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.704022    5031 scope.go:122] "RemoveContainer" containerID="fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81"
Oct 30 22:25:14.704333 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:14.704311    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81\": container with ID starting with fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81 not found: ID does not exist" containerID="fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81"
Oct 30 22:25:14.704410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.704344    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81"} err="failed to get container status \"fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81\": rpc error: code = NotFound desc = could not find container \"fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81\": container with ID starting with fd83a190355cb43bfb56aaa24f2804a8fe447d168884922a8d264dbef20a9e81 not found: ID does not exist"
Oct 30 22:25:14.851060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.851023    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"]
Oct 30 22:25:14.859354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.859317    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/premium-w4-bce513-kserve-798dd849b5-54kfv"]
Oct 30 22:25:14.872564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.871665    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:14.970505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.970453    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"]
Oct 30 22:25:14.974595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:14.974564    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/simulator-w4-bce513-kserve-695578ddd8-mg4tz"]
Oct 30 22:25:15.089073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:15.089038    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"]
Oct 30 22:25:15.093188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:15.093118    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bce513/distinct-2-w4-bce513-kserve-fd56f47ff-5wb4w"]
Oct 30 22:25:15.135285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:15.135256    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:15.881402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:15.881362    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:16.648553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:16.648518    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" path="/var/lib/kubelet/pods/1241795d-784e-4ab5-8b92-66d23a747f3e/volumes"
Oct 30 22:25:16.649001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:16.648988    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27147da-5731-4444-92d8-58f799907b2b" path="/var/lib/kubelet/pods/b27147da-5731-4444-92d8-58f799907b2b/volumes"
Oct 30 22:25:16.649424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:16.649410    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" path="/var/lib/kubelet/pods/ff7a540c-99bf-420d-a7eb-ef1664d01bce/volumes"
Oct 30 22:25:17.260200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:17.260160    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" podUID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:20.487556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:20.487514    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:20.546418 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:20.546384    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:25:20.764041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:20.763947    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.031400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:22.031362    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:24.858995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:24.858952    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"]
Oct 30 22:25:24.860096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:24.859171    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" containerID="cri-o://f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3" gracePeriod=30
Oct 30 22:25:24.908486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:24.908441    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"]
Oct 30 22:25:24.908740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:24.908711    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" containerID="cri-o://2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1" gracePeriod=30
Oct 30 22:25:25.733060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:25.733016    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"]
Oct 30 22:25:25.776182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:25.776144    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" containerID="cri-o://79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0" gracePeriod=30
Oct 30 22:25:25.960847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:25.960811    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:25.961275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:25.960881    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:26.638867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:26.638823    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:26.639113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:26.638890    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:26.699755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:26.699715    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:26.699947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:26.699777    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:27.260057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:27.260015    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" podUID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:27.444298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:27.444197    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:28.670736 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:28.670659    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:28.973361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:28.973260    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:29.956590 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:29.956128    5031 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff1d206_820a_41f9_85e0_da237ca641ff.slice/crio-f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:25:30.077298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.076532    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:25:30.183889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.183743    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:25:30.211252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.206314    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-kube-api-access-nz4fg\" (UniqueName: \"kubernetes.io/projected/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-kube-api-access-nz4fg\") pod \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") "
Oct 30 22:25:30.211252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.206377    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-plugins-config-volume\") pod \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\" (UID: \"730e9a61-0bf3-44bf-bd86-3cad576ab1d0\") "
Oct 30 22:25:30.211252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.207901    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-plugins-config-volume" pod "730e9a61-0bf3-44bf-bd86-3cad576ab1d0" (UID: "730e9a61-0bf3-44bf-bd86-3cad576ab1d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:30.211252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.210615    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-kube-api-access-nz4fg" pod "730e9a61-0bf3-44bf-bd86-3cad576ab1d0" (UID: "730e9a61-0bf3-44bf-bd86-3cad576ab1d0"). InnerVolumeSpecName "kube-api-access-nz4fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:30.237638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.237609    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:25:30.308260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308163    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff1d206-820a-41f9-85e0-da237ca641ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1d206-820a-41f9-85e0-da237ca641ff-istiod-ca-cert\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.308260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308247    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-credential-socket\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.308513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308331    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-envoy\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.308513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308391    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-credential-socket\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.308513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308411    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff1d206-820a-41f9-85e0-da237ca641ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1d206-820a-41f9-85e0-da237ca641ff-istio-podinfo\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.308513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308435    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-socket\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.308513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308461    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-istio-token\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-istio-token\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.308513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308486    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-certs\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.308841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308555    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-credential-socket" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.308841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308566    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-certs\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.308841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308612    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff1d206-820a-41f9-85e0-da237ca641ff-istiod-ca-cert" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:30.308841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308722    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-kube-api-access-d47zd\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-kube-api-access-d47zd\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.308841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308779    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-socket\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.308841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308819    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-credential-socket" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308861    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69863d8f-0231-4dd2-ae46-1d13bc939df3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69863d8f-0231-4dd2-ae46-1d13bc939df3-istiod-ca-cert\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308896    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-data\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308907    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-socket" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308935    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-envoy\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308954    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-certs" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.308973    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-data\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309020    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-kube-api-access-cd97b\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-kube-api-access-cd97b\") pod \"aff1d206-820a-41f9-85e0-da237ca641ff\" (UID: \"aff1d206-820a-41f9-85e0-da237ca641ff\") "
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309043    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-token\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-token\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309064    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-certs" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.309138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309073    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-podinfo\") pod \"69863d8f-0231-4dd2-ae46-1d13bc939df3\" (UID: \"69863d8f-0231-4dd2-ae46-1d13bc939df3\") "
Oct 30 22:25:30.309904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309306    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-socket" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.309904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309352    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69863d8f-0231-4dd2-ae46-1d13bc939df3-istiod-ca-cert" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:30.309904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309541    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-data" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.309904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309595    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-data" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.309994    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69863d8f-0231-4dd2-ae46-1d13bc939df3-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310035    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310050    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310067    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz4fg\" (UniqueName: \"kubernetes.io/projected/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-kube-api-access-nz4fg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310085    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730e9a61-0bf3-44bf-bd86-3cad576ab1d0-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310099    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff1d206-820a-41f9-85e0-da237ca641ff-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310114    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310128    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310142    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310156    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310170    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.310704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.310183    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.311917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.311878    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-envoy" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.312078 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.312032    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-istio-token" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:30.312159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.312133    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-podinfo" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:30.312305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.312288    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-envoy" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:30.312593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.312575    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-kube-api-access-cd97b" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "kube-api-access-cd97b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:30.312668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.312591    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-kube-api-access-d47zd" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "kube-api-access-d47zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:30.312726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.312685    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff1d206-820a-41f9-85e0-da237ca641ff-istio-podinfo" pod "aff1d206-820a-41f9-85e0-da237ca641ff" (UID: "aff1d206-820a-41f9-85e0-da237ca641ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:30.312949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.312934    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-token" pod "69863d8f-0231-4dd2-ae46-1d13bc939df3" (UID: "69863d8f-0231-4dd2-ae46-1d13bc939df3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:30.411958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.411917    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff1d206-820a-41f9-85e0-da237ca641ff-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.411958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.411949    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd97b\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-kube-api-access-cd97b\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.411958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.411960    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.412327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.411969    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.412327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.411979    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69863d8f-0231-4dd2-ae46-1d13bc939df3-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.412327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.411988    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff1d206-820a-41f9-85e0-da237ca641ff-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.412327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.411996    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff1d206-820a-41f9-85e0-da237ca641ff-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.412327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.412005    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47zd\" (UniqueName: \"kubernetes.io/projected/69863d8f-0231-4dd2-ae46-1d13bc939df3-kube-api-access-d47zd\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.821491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.821456    5031 generic.go:356] "Generic (PLEG): container finished" podID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" containerID="4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9" exitCode=0
Oct 30 22:25:30.821637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.821484    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" event={"ID":"730e9a61-0bf3-44bf-bd86-3cad576ab1d0","Type":"ContainerDied","Data":"4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9"}
Oct 30 22:25:30.821637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.821497    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"
Oct 30 22:25:30.821637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.821545    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787" event={"ID":"730e9a61-0bf3-44bf-bd86-3cad576ab1d0","Type":"ContainerDied","Data":"0a3c6dc074d212d4b687c205f487ec72eae47a4c00965982ca0bd51f78df04d5"}
Oct 30 22:25:30.821637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.821570    5031 scope.go:122] "RemoveContainer" containerID="4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9"
Oct 30 22:25:30.823271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.823247    5031 generic.go:356] "Generic (PLEG): container finished" podID="aff1d206-820a-41f9-85e0-da237ca641ff" containerID="f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3" exitCode=0
Oct 30 22:25:30.823271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.823266    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"
Oct 30 22:25:30.823418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.823317    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" event={"ID":"aff1d206-820a-41f9-85e0-da237ca641ff","Type":"ContainerDied","Data":"f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3"}
Oct 30 22:25:30.823418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.823362    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh" event={"ID":"aff1d206-820a-41f9-85e0-da237ca641ff","Type":"ContainerDied","Data":"ad4c91c8667bfeaf6e3b4cb61cd6e0dd796e303a59653e919e357663f3ace95b"}
Oct 30 22:25:30.824973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.824953    5031 generic.go:356] "Generic (PLEG): container finished" podID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerID="2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1" exitCode=0
Oct 30 22:25:30.825062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.824986    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" event={"ID":"69863d8f-0231-4dd2-ae46-1d13bc939df3","Type":"ContainerDied","Data":"2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1"}
Oct 30 22:25:30.825062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.825001    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"
Oct 30 22:25:30.825062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.825033    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg" event={"ID":"69863d8f-0231-4dd2-ae46-1d13bc939df3","Type":"ContainerDied","Data":"e50906739273e3ce527ee92536fe59eb5332dd67d0ce584d4c834d37b1f48805"}
Oct 30 22:25:30.891341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.891163    5031 scope.go:122] "RemoveContainer" containerID="4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9"
Oct 30 22:25:30.891632 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:30.891610    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9\": container with ID starting with 4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9 not found: ID does not exist" containerID="4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9"
Oct 30 22:25:30.891717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.891643    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9"} err="failed to get container status \"4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9\": rpc error: code = NotFound desc = could not find container \"4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9\": container with ID starting with 4e6296ad789f57773dc92696f087a07b02f4c351d39120149f528c9e60f708f9 not found: ID does not exist"
Oct 30 22:25:30.891717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.891661    5031 scope.go:122] "RemoveContainer" containerID="f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3"
Oct 30 22:25:30.902247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.902223    5031 scope.go:122] "RemoveContainer" containerID="f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3"
Oct 30 22:25:30.902540 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:30.902520    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3\": container with ID starting with f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3 not found: ID does not exist" containerID="f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3"
Oct 30 22:25:30.902654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.902552    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3"} err="failed to get container status \"f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3\": rpc error: code = NotFound desc = could not find container \"f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3\": container with ID starting with f10ff3f83c706679f19481e7013e0bbe2b66f340abecf6021c01d9abb1d72fb3 not found: ID does not exist"
Oct 30 22:25:30.902654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.902573    5031 scope.go:122] "RemoveContainer" containerID="2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1"
Oct 30 22:25:30.913188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.913166    5031 scope.go:122] "RemoveContainer" containerID="2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1"
Oct 30 22:25:30.913547 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:30.913483    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1\": container with ID starting with 2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1 not found: ID does not exist" containerID="2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1"
Oct 30 22:25:30.913547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.913512    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1"} err="failed to get container status \"2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1\": rpc error: code = NotFound desc = could not find container \"2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1\": container with ID starting with 2ad9ad8ac98a7feb4148d23d2a0baeb3feb5039373b66aa63850f6be6471f3f1 not found: ID does not exist"
Oct 30 22:25:30.998408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:30.998370    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"]
Oct 30 22:25:31.008626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.008589    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787"]
Oct 30 22:25:31.053782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.053754    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:25:31.120466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120433    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-token\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-token\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.120650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120475    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-data\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.120650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120497    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-socket\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.120650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120573    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-credential-socket\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.120650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120607    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-envoy\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.120889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120685    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-podinfo\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.120889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120727    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-certs\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.120889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120756    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-data" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.121060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120893    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-kube-api-access-ctfm8\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-kube-api-access-ctfm8\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.121060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120913    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-credential-socket" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.121060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120935    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-socket" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.121060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.120965    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/249019e3-0f9c-4671-9b57-d629ee30cd6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249019e3-0f9c-4671-9b57-d629ee30cd6b-istiod-ca-cert\") pod \"249019e3-0f9c-4671-9b57-d629ee30cd6b\" (UID: \"249019e3-0f9c-4671-9b57-d629ee30cd6b\") "
Oct 30 22:25:31.121060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.121054    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-certs" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.121408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.121386    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249019e3-0f9c-4671-9b57-d629ee30cd6b-istiod-ca-cert" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:31.121663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.121615    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.121663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.121631    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.121663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.121641    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249019e3-0f9c-4671-9b57-d629ee30cd6b-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.121663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.121651    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.121663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.121659    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.123001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.122975    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-token" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:31.123406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.123389    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-kube-api-access-ctfm8" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "kube-api-access-ctfm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:31.123574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.123558    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-envoy" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:31.123624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.123586    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-podinfo" pod "249019e3-0f9c-4671-9b57-d629ee30cd6b" (UID: "249019e3-0f9c-4671-9b57-d629ee30cd6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:31.133462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.133409    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"]
Oct 30 22:25:31.150141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.150108    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg"]
Oct 30 22:25:31.223554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.223471    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfm8\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-kube-api-access-ctfm8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.223554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.223503    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.223554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.223517    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.223554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.223529    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249019e3-0f9c-4671-9b57-d629ee30cd6b-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.277622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.277577    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"]
Oct 30 22:25:31.279231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.279181    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh"]
Oct 30 22:25:31.838089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.838055    5031 generic.go:356] "Generic (PLEG): container finished" podID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerID="79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0" exitCode=0
Oct 30 22:25:31.838346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.838131    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"
Oct 30 22:25:31.838346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.838132    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" event={"ID":"249019e3-0f9c-4671-9b57-d629ee30cd6b","Type":"ContainerDied","Data":"79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0"}
Oct 30 22:25:31.838346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.838201    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr" event={"ID":"249019e3-0f9c-4671-9b57-d629ee30cd6b","Type":"ContainerDied","Data":"c87ef66d3a5076f597aba7b42b6a67dc817bc8e90d465c76a99752ae4bdce144"}
Oct 30 22:25:31.838346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.838256    5031 scope.go:122] "RemoveContainer" containerID="79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0"
Oct 30 22:25:31.850566 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.850547    5031 scope.go:122] "RemoveContainer" containerID="79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0"
Oct 30 22:25:31.850858 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:31.850835    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0\": container with ID starting with 79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0 not found: ID does not exist" containerID="79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0"
Oct 30 22:25:31.850943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:31.850864    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0"} err="failed to get container status \"79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0\": rpc error: code = NotFound desc = could not find container \"79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0\": container with ID starting with 79bca47b19eff6a3d751820a4509616f652aa1ff37b8f2363ea748f79821e0f0 not found: ID does not exist"
Oct 30 22:25:32.024170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:32.024127    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"]
Oct 30 22:25:32.030762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:32.030731    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr"]
Oct 30 22:25:32.419475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:32.419432    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:32.647329 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:32.647293    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" path="/var/lib/kubelet/pods/249019e3-0f9c-4671-9b57-d629ee30cd6b/volumes"
Oct 30 22:25:32.647819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:32.647803    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" path="/var/lib/kubelet/pods/69863d8f-0231-4dd2-ae46-1d13bc939df3/volumes"
Oct 30 22:25:32.648308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:32.648291    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" path="/var/lib/kubelet/pods/730e9a61-0bf3-44bf-bd86-3cad576ab1d0/volumes"
Oct 30 22:25:32.649055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:32.649039    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" path="/var/lib/kubelet/pods/aff1d206-820a-41f9-85e0-da237ca641ff/volumes"
Oct 30 22:25:33.673936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:33.673892    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.079429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:34.079346    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.373419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:34.373316    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.620257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:34.620201    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"]
Oct 30 22:25:34.668708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:34.668623    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-516ccb-njp6n"]
Oct 30 22:25:35.142644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:35.142612    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.403516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:35.403414    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.764872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:35.764767    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.958132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:35.958095    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"]
Oct 30 22:25:35.979348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:35.979300    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7cfd50-2r5rc"]
Oct 30 22:25:36.038459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:36.038370    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:36.519142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:36.519108    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:36.647355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:36.647318    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8" path="/var/lib/kubelet/pods/d5ac7763-9f61-4dc5-8e7f-7d24b90bcae8/volumes"
Oct 30 22:25:36.647833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:36.647817    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc" path="/var/lib/kubelet/pods/ddaac269-12f8-4ca2-a1d1-b3ab41f3a4bc/volumes"
Oct 30 22:25:41.487390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:41.487348    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:43.477741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:43.477700    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:43.811963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:43.811861    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:44.060512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.060480    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:44.637121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637087    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"]
Oct 30 22:25:44.637761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637744    5031 cpu_manager.go:414] "containerMap: removing container" podUID="eb469053-f5e2-4e90-9f98-be803e987d25" containerName="revoke-keys"
Oct 30 22:25:44.637838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637761    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb469053-f5e2-4e90-9f98-be803e987d25" containerName="revoke-keys"
Oct 30 22:25:44.637838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637786    5031 cpu_manager.go:414] "containerMap: removing container" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy"
Oct 30 22:25:44.637838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637792    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy"
Oct 30 22:25:44.637838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637807    5031 cpu_manager.go:414] "containerMap: removing container" podUID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" containerName="payload-processing"
Oct 30 22:25:44.637838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637812    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" containerName="payload-processing"
Oct 30 22:25:44.637838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637822    5031 cpu_manager.go:414] "containerMap: removing container" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy"
Oct 30 22:25:44.637838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637826    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637845    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637850    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637862    5031 cpu_manager.go:414] "containerMap: removing container" podUID="414303a8-eacb-4472-8115-f8be154c96d1" containerName="revoke-keys"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637866    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414303a8-eacb-4472-8115-f8be154c96d1" containerName="revoke-keys"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637879    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637883    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637891    5031 cpu_manager.go:414] "containerMap: removing container" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637895    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637904    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637908    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.637999    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff1d206-820a-41f9-85e0-da237ca641ff" containerName="istio-proxy"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638010    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="730e9a61-0bf3-44bf-bd86-3cad576ab1d0" containerName="payload-processing"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638020    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="249019e3-0f9c-4671-9b57-d629ee30cd6b" containerName="istio-proxy"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638032    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1241795d-784e-4ab5-8b92-66d23a747f3e" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638044    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="414303a8-eacb-4472-8115-f8be154c96d1" containerName="revoke-keys"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638053    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27147da-5731-4444-92d8-58f799907b2b" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638061    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="69863d8f-0231-4dd2-ae46-1d13bc939df3" containerName="istio-proxy"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638070    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7a540c-99bf-420d-a7eb-ef1664d01bce" containerName="main"
Oct 30 22:25:44.638233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.638079    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb469053-f5e2-4e90-9f98-be803e987d25" containerName="revoke-keys"
Oct 30 22:25:44.642081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.642049    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.700414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.700375    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.700698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.700669    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dn6\" (UniqueName: \"kubernetes.io/projected/c82c5bca-f443-43da-9df4-030c1b7aa6bb-kube-api-access-b7dn6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.700827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.700743    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.726235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.726194    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:44.757686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.757645    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:25:44.757686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.757653    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:44.757918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.757724    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:44.757918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.757804    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:44.761783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.761761    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"]
Oct 30 22:25:44.801860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.801829    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dn6\" (UniqueName: \"kubernetes.io/projected/c82c5bca-f443-43da-9df4-030c1b7aa6bb-kube-api-access-b7dn6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.801994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.801890    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.801994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.801966    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.802754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.802729    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.804549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.804519    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.871877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.871850    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dn6\" (UniqueName: \"kubernetes.io/projected/c82c5bca-f443-43da-9df4-030c1b7aa6bb-kube-api-access-b7dn6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.953289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.949447    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"]
Oct 30 22:25:44.953289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.949792    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy" containerID="cri-o://f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3" gracePeriod=30
Oct 30 22:25:44.960318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.960290    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:44.966539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:44.966504    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:45.192098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:45.192046    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:45.359525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:45.359494    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"]
Oct 30 22:25:45.360703 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:25:45.360671    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82c5bca_f443_43da_9df4_030c1b7aa6bb.slice/crio-cb2c8cc4814987a950cfac52cdc1f176a70d8e0aea1adb2670ceaa2d1364d6cd WatchSource:0}: Error finding container cb2c8cc4814987a950cfac52cdc1f176a70d8e0aea1adb2670ceaa2d1364d6cd: Status 404 returned error can't find the container with id cb2c8cc4814987a950cfac52cdc1f176a70d8e0aea1adb2670ceaa2d1364d6cd
Oct 30 22:25:45.883703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:45.883670    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:45.983316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:45.983222    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl" event={"ID":"c82c5bca-f443-43da-9df4-030c1b7aa6bb","Type":"ContainerStarted","Data":"8c2399ba11874bd56c9a50f802046f120ae89139b702128f6a8d28cff0ba43ca"}
Oct 30 22:25:45.983316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:45.983287    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl" event={"ID":"c82c5bca-f443-43da-9df4-030c1b7aa6bb","Type":"ContainerStarted","Data":"cb2c8cc4814987a950cfac52cdc1f176a70d8e0aea1adb2670ceaa2d1364d6cd"}
Oct 30 22:25:46.106601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.106547    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl" podStartSLOduration=2.106531405 podStartE2EDuration="2.106531405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:46.102503374 +0000 UTC m=+2157.626646161" watchObservedRunningTime="2026-10-30 22:25:46.106531405 +0000 UTC m=+2157.630674169"
Oct 30 22:25:46.370857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.370817    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"]
Oct 30 22:25:46.395493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.390904    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.438589    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-envoy\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.438638    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-certs\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.438699    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr7g\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-kube-api-access-lnr7g\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.438811    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istiod-ca-cert\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.438837    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-podinfo\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.438866    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-socket\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.438911    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-token\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.439667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.439088    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-data\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.446243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.439930    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-credential-socket\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.448947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.448916    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"]
Oct 30 22:25:46.567283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.566680    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:46.571542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.571452    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d67bf9-openshift-default-dockercfg-wj6cz\""
Oct 30 22:25:46.576969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.576935    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-envoy\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.577174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.576987    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-certs\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.577174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.577049    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr7g\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-kube-api-access-lnr7g\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.577174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.577158    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istiod-ca-cert\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.577399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.577192    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-podinfo\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.583021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.582960    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-certs\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.583389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.583365    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-socket\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.583494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.583431    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-token\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.583555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.583536    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-data\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.583617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.583584    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-credential-socket\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.584017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.583994    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-socket\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.584999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.584916    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istiod-ca-cert\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.586623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.586596    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-data\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.587797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.587762    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-credential-socket\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.595433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.595409    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-envoy\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.599504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.599477    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-podinfo\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.648927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.648896    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-token\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.649513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.649486    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr7g\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-kube-api-access-lnr7g\") pod \"e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") " pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.750949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.750904    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:46.908618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.908578    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:46.993319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.993279    5031 generic.go:356] "Generic (PLEG): container finished" podID="c82c5bca-f443-43da-9df4-030c1b7aa6bb" containerID="8c2399ba11874bd56c9a50f802046f120ae89139b702128f6a8d28cff0ba43ca" exitCode=0
Oct 30 22:25:46.993530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:46.993360    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl" event={"ID":"c82c5bca-f443-43da-9df4-030c1b7aa6bb","Type":"ContainerDied","Data":"8c2399ba11874bd56c9a50f802046f120ae89139b702128f6a8d28cff0ba43ca"}
Oct 30 22:25:47.232866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:47.232828    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"]
Oct 30 22:25:47.234136 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:25:47.234102    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d4ccaf_f1e8_4aae_8351_98ff0b3f9014.slice/crio-9b1ac52d578e9204f421ef68fd1102ecb4789f3606924c1165d0a7c6fc73efd6 WatchSource:0}: Error finding container 9b1ac52d578e9204f421ef68fd1102ecb4789f3606924c1165d0a7c6fc73efd6: Status 404 returned error can't find the container with id 9b1ac52d578e9204f421ef68fd1102ecb4789f3606924c1165d0a7c6fc73efd6
Oct 30 22:25:47.237786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:47.237727    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:25:47.237894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:47.237832    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:25:47.237894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:47.237875    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:25:47.501688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:47.501607    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:48.003478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.003435    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" event={"ID":"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014","Type":"ContainerStarted","Data":"ff5c4f8d7cddf333b84b76d79bc77d9fdfcc44f69356970f8943dfcfff997581"}
Oct 30 22:25:48.003954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.003503    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" event={"ID":"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014","Type":"ContainerStarted","Data":"9b1ac52d578e9204f421ef68fd1102ecb4789f3606924c1165d0a7c6fc73efd6"}
Oct 30 22:25:48.135718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.135646    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podStartSLOduration=2.135625276 podStartE2EDuration="2.135625276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:48.134089777 +0000 UTC m=+2159.658232543" watchObservedRunningTime="2026-10-30 22:25:48.135625276 +0000 UTC m=+2159.659768041"
Oct 30 22:25:48.208054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.208016    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:48.308070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.307974    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-openshift-ca\") pod \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") "
Oct 30 22:25:48.308240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.308182    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82c5bca-f443-43da-9df4-030c1b7aa6bb-kube-api-access-b7dn6\" (UniqueName: \"kubernetes.io/projected/c82c5bca-f443-43da-9df4-030c1b7aa6bb-kube-api-access-b7dn6\") pod \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") "
Oct 30 22:25:48.308335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.308293    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-xks-ca\") pod \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\" (UID: \"c82c5bca-f443-43da-9df4-030c1b7aa6bb\") "
Oct 30 22:25:48.308400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.308368    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-openshift-ca" pod "c82c5bca-f443-43da-9df4-030c1b7aa6bb" (UID: "c82c5bca-f443-43da-9df4-030c1b7aa6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:48.309008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.308922    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:48.310642 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.310617    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-xks-ca" pod "c82c5bca-f443-43da-9df4-030c1b7aa6bb" (UID: "c82c5bca-f443-43da-9df4-030c1b7aa6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:48.310743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.310639    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82c5bca-f443-43da-9df4-030c1b7aa6bb-kube-api-access-b7dn6" pod "c82c5bca-f443-43da-9df4-030c1b7aa6bb" (UID: "c82c5bca-f443-43da-9df4-030c1b7aa6bb"). InnerVolumeSpecName "kube-api-access-b7dn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:48.409870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.409839    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7dn6\" (UniqueName: \"kubernetes.io/projected/c82c5bca-f443-43da-9df4-030c1b7aa6bb-kube-api-access-b7dn6\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:48.409870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.409870    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82c5bca-f443-43da-9df4-030c1b7aa6bb-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:48.751913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.751883    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:48.758883 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:48.758854    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:49.013526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:49.013426    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"
Oct 30 22:25:49.013526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:49.013462    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl" event={"ID":"c82c5bca-f443-43da-9df4-030c1b7aa6bb","Type":"ContainerDied","Data":"cb2c8cc4814987a950cfac52cdc1f176a70d8e0aea1adb2670ceaa2d1364d6cd"}
Oct 30 22:25:49.013526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:49.013516    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2c8cc4814987a950cfac52cdc1f176a70d8e0aea1adb2670ceaa2d1364d6cd"
Oct 30 22:25:49.014152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:49.013948    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:49.015159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:49.015134    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:25:50.231695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.231632    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:25:50.344358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344318    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-kube-api-access-72t2g\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-kube-api-access-72t2g\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344389    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-data\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344416    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-credential-socket\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344461    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-podinfo\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344494    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-certs\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344533    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-token\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-token\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344567    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-envoy\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344730    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istiod-ca-cert\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.344845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344805    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-data" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:50.344845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344814    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-socket\") pod \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\" (UID: \"f6fcb1e3-b93e-42e8-b489-bbec1d3409de\") "
Oct 30 22:25:50.345068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.344838    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-credential-socket" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:50.345068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345032    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-socket" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:50.345300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345270    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istiod-ca-cert" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:50.345300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345283    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-certs" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:50.345663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345629    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.345663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345654    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.345874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345669    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.345874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345684    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.345874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.345698    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.347380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.347354    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-envoy" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:50.347514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.347403    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-kube-api-access-72t2g" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "kube-api-access-72t2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:50.347766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.347745    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-token" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:50.347946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.347921    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-podinfo" pod "f6fcb1e3-b93e-42e8-b489-bbec1d3409de" (UID: "f6fcb1e3-b93e-42e8-b489-bbec1d3409de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:50.448536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.448495    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72t2g\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-kube-api-access-72t2g\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.448536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.448538    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.448795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.448554    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.448795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:50.448567    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fcb1e3-b93e-42e8-b489-bbec1d3409de-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:50.544560 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:50.544468    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:25:51.036912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.036873    5031 generic.go:356] "Generic (PLEG): container finished" podID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerID="f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3" exitCode=0
Oct 30 22:25:51.037151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.036939    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"
Oct 30 22:25:51.037151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.036951    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" event={"ID":"f6fcb1e3-b93e-42e8-b489-bbec1d3409de","Type":"ContainerDied","Data":"f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3"}
Oct 30 22:25:51.037151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.037031    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm" event={"ID":"f6fcb1e3-b93e-42e8-b489-bbec1d3409de","Type":"ContainerDied","Data":"1bd66b7898fe4f234c13871eb9707f5f089676d04fb99dbaafcce2c9355c1fd9"}
Oct 30 22:25:51.037151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.037065    5031 scope.go:122] "RemoveContainer" containerID="f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3"
Oct 30 22:25:51.061898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.061863    5031 scope.go:122] "RemoveContainer" containerID="b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd"
Oct 30 22:25:51.074958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.074730    5031 scope.go:122] "RemoveContainer" containerID="f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3"
Oct 30 22:25:51.075082 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:51.075059    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3\": container with ID starting with f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3 not found: ID does not exist" containerID="f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3"
Oct 30 22:25:51.075170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.075092    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3"} err="failed to get container status \"f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3\": rpc error: code = NotFound desc = could not find container \"f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3\": container with ID starting with f82078a88e7e3202627551637f40c328e9e7790ea4547ed96d60ddffc2a431c3 not found: ID does not exist"
Oct 30 22:25:51.075170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.075114    5031 scope.go:122] "RemoveContainer" containerID="b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd"
Oct 30 22:25:51.075469 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:25:51.075444    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd\": container with ID starting with b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd not found: ID does not exist" containerID="b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd"
Oct 30 22:25:51.075528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.075473    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd"} err="failed to get container status \"b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd\": rpc error: code = NotFound desc = could not find container \"b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd\": container with ID starting with b6a68a1395f8ca9c5c031958b3689063d9490edf68f01c5e8c5833dfcf8fbefd not found: ID does not exist"
Oct 30 22:25:51.088338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.088269    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.263778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.263742    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"]
Oct 30 22:25:51.267546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.267519    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm"]
Oct 30 22:25:51.332154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.332032    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.348150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.348116    5031 scope.go:122] "RemoveContainer" containerID="e12e7948309001d1774a37b9c7d01c08e292fc47f3f4906a89475468c9fbbbf1"
Oct 30 22:25:51.564161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.564121    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.971025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:51.970976    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:25:52.223304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:52.223192    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.649579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:52.649537    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" path="/var/lib/kubelet/pods/f6fcb1e3-b93e-42e8-b489-bbec1d3409de/volumes"
Oct 30 22:25:54.329331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.329301    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.608660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.608620    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.832476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.832437    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"]
Oct 30 22:25:54.833086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833073    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy"
Oct 30 22:25:54.833142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833088    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy"
Oct 30 22:25:54.833142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833123    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c82c5bca-f443-43da-9df4-030c1b7aa6bb" containerName="revoke-keys"
Oct 30 22:25:54.833142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833128    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82c5bca-f443-43da-9df4-030c1b7aa6bb" containerName="revoke-keys"
Oct 30 22:25:54.833142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833134    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy"
Oct 30 22:25:54.833142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833139    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy"
Oct 30 22:25:54.833337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833250    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy"
Oct 30 22:25:54.833337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.833265    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82c5bca-f443-43da-9df4-030c1b7aa6bb" containerName="revoke-keys"
Oct 30 22:25:54.839042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.839021    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:54.858456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.858429    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.894184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.894087    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d67bf9-dockercfg-wxjp4\""
Oct 30 22:25:54.894660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.894630    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d67bf9\""
Oct 30 22:25:54.897725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.897688    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"]
Oct 30 22:25:54.992236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.992179    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fj5f\" (UniqueName: \"kubernetes.io/projected/730923a4-658d-4e33-904d-9c475ed2f828-kube-api-access-8fj5f\") pod \"payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:54.992757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:54.992728    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730923a4-658d-4e33-904d-9c475ed2f828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:55.001335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.001295    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"]
Oct 30 22:25:55.006904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.006875    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fcb1e3-b93e-42e8-b489-bbec1d3409de" containerName="istio-proxy"
Oct 30 22:25:55.014270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.011879    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.029675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.029439    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"]
Oct 30 22:25:55.094675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.094641    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fj5f\" (UniqueName: \"kubernetes.io/projected/730923a4-658d-4e33-904d-9c475ed2f828-kube-api-access-8fj5f\") pod \"payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:55.094877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.094801    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730923a4-658d-4e33-904d-9c475ed2f828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:55.095593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.095564    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730923a4-658d-4e33-904d-9c475ed2f828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:55.167573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.167488    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fj5f\" (UniqueName: \"kubernetes.io/projected/730923a4-658d-4e33-904d-9c475ed2f828-kube-api-access-8fj5f\") pod \"payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:55.196620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.196574    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d5e446-9c52-4f08-bd66-7d04ec22899b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.196784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.196718    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6p88\" (UniqueName: \"kubernetes.io/projected/32d5e446-9c52-4f08-bd66-7d04ec22899b-kube-api-access-m6p88\") pod \"payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.297772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.297734    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d5e446-9c52-4f08-bd66-7d04ec22899b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.297965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.297831    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6p88\" (UniqueName: \"kubernetes.io/projected/32d5e446-9c52-4f08-bd66-7d04ec22899b-kube-api-access-m6p88\") pod \"payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.298589 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.298568    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d5e446-9c52-4f08-bd66-7d04ec22899b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.362445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.362409    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6p88\" (UniqueName: \"kubernetes.io/projected/32d5e446-9c52-4f08-bd66-7d04ec22899b-kube-api-access-m6p88\") pod \"payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") " pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.453113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.453023    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:55.637763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.637724    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:55.670959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.670914    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:55.840934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.840902    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"]
Oct 30 22:25:55.841444 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:25:55.841408    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730923a4_658d_4e33_904d_9c475ed2f828.slice/crio-8b4dc15ea0410748873599a1a34a8c10be2704215588e986c748e8b4921e8bd3 WatchSource:0}: Error finding container 8b4dc15ea0410748873599a1a34a8c10be2704215588e986c748e8b4921e8bd3: Status 404 returned error can't find the container with id 8b4dc15ea0410748873599a1a34a8c10be2704215588e986c748e8b4921e8bd3
Oct 30 22:25:55.943359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:55.943313    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.025522 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:25:56.025479    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d5e446_9c52_4f08_bd66_7d04ec22899b.slice/crio-7d6a0f466c075e4cf4d2d305582f6fc0349fb490d4a7bc0b8a6e759cedf5c5f0 WatchSource:0}: Error finding container 7d6a0f466c075e4cf4d2d305582f6fc0349fb490d4a7bc0b8a6e759cedf5c5f0: Status 404 returned error can't find the container with id 7d6a0f466c075e4cf4d2d305582f6fc0349fb490d4a7bc0b8a6e759cedf5c5f0
Oct 30 22:25:56.025776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.025758    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"]
Oct 30 22:25:56.100381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.100353    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2" event={"ID":"730923a4-658d-4e33-904d-9c475ed2f828","Type":"ContainerStarted","Data":"7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216"}
Oct 30 22:25:56.100507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.100404    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2" event={"ID":"730923a4-658d-4e33-904d-9c475ed2f828","Type":"ContainerStarted","Data":"8b4dc15ea0410748873599a1a34a8c10be2704215588e986c748e8b4921e8bd3"}
Oct 30 22:25:56.100582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.100567    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:25:56.102303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.102265    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz" event={"ID":"32d5e446-9c52-4f08-bd66-7d04ec22899b","Type":"ContainerStarted","Data":"1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee"}
Oct 30 22:25:56.102412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.102320    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:25:56.102412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.102338    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz" event={"ID":"32d5e446-9c52-4f08-bd66-7d04ec22899b","Type":"ContainerStarted","Data":"7d6a0f466c075e4cf4d2d305582f6fc0349fb490d4a7bc0b8a6e759cedf5c5f0"}
Oct 30 22:25:56.109630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.109605    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"]
Oct 30 22:25:56.113895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.113873    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24aa4c-qhnwt"]
Oct 30 22:25:56.222304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.222244    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2" podStartSLOduration=2.222227345 podStartE2EDuration="2.222227345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:56.219462237 +0000 UTC m=+2167.743605002" watchObservedRunningTime="2026-10-30 22:25:56.222227345 +0000 UTC m=+2167.746370106"
Oct 30 22:25:56.352632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.352558    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz" podStartSLOduration=2.352537077 podStartE2EDuration="2.352537077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:56.348029703 +0000 UTC m=+2167.872172484" watchObservedRunningTime="2026-10-30 22:25:56.352537077 +0000 UTC m=+2167.876679841"
Oct 30 22:25:56.597665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.597627    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.647188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:56.647105    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e359e3-5dd3-429f-9d5a-1d75e5d23473" path="/var/lib/kubelet/pods/09e359e3-5dd3-429f-9d5a-1d75e5d23473/volumes"
Oct 30 22:25:57.164121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:57.164088    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"]
Oct 30 22:25:57.503093 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:57.502969    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:57.757664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:25:57.757585    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:00.082099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:00.082068    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.164945 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.164873    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4" podUID="7787089a-e528-4c5c-b083-dc3d94fb6db4" containerName="payload-pre-processing" containerID="cri-o://ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958" gracePeriod=25
Oct 30 22:26:02.453575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.453549    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:26:02.502460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.502418    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7787089a-e528-4c5c-b083-dc3d94fb6db4-kube-api-access-p8pbc\" (UniqueName: \"kubernetes.io/projected/7787089a-e528-4c5c-b083-dc3d94fb6db4-kube-api-access-p8pbc\") pod \"7787089a-e528-4c5c-b083-dc3d94fb6db4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") "
Oct 30 22:26:02.502677 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.502501    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7787089a-e528-4c5c-b083-dc3d94fb6db4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7787089a-e528-4c5c-b083-dc3d94fb6db4-plugins-config-volume\") pod \"7787089a-e528-4c5c-b083-dc3d94fb6db4\" (UID: \"7787089a-e528-4c5c-b083-dc3d94fb6db4\") "
Oct 30 22:26:02.502891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.502859    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7787089a-e528-4c5c-b083-dc3d94fb6db4-plugins-config-volume" pod "7787089a-e528-4c5c-b083-dc3d94fb6db4" (UID: "7787089a-e528-4c5c-b083-dc3d94fb6db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:02.503021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.502977    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7787089a-e528-4c5c-b083-dc3d94fb6db4-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.504971 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.504947    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7787089a-e528-4c5c-b083-dc3d94fb6db4-kube-api-access-p8pbc" pod "7787089a-e528-4c5c-b083-dc3d94fb6db4" (UID: "7787089a-e528-4c5c-b083-dc3d94fb6db4"). InnerVolumeSpecName "kube-api-access-p8pbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:02.604190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:02.604154    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8pbc\" (UniqueName: \"kubernetes.io/projected/7787089a-e528-4c5c-b083-dc3d94fb6db4-kube-api-access-p8pbc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:03.165084 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.165051    5031 generic.go:356] "Generic (PLEG): container finished" podID="7787089a-e528-4c5c-b083-dc3d94fb6db4" containerID="ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958" exitCode=0
Oct 30 22:26:03.165599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.165102    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"
Oct 30 22:26:03.165599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.165135    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4" event={"ID":"7787089a-e528-4c5c-b083-dc3d94fb6db4","Type":"ContainerDied","Data":"ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958"}
Oct 30 22:26:03.165599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.165189    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4" event={"ID":"7787089a-e528-4c5c-b083-dc3d94fb6db4","Type":"ContainerDied","Data":"22f8dac918f3603e75e22ef7ecad5a73ca392d0ca0e1b6013a74603f638b5029"}
Oct 30 22:26:03.165599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.165243    5031 scope.go:122] "RemoveContainer" containerID="ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958"
Oct 30 22:26:03.178463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.178441    5031 scope.go:122] "RemoveContainer" containerID="ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958"
Oct 30 22:26:03.178830 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:26:03.178801    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958\": container with ID starting with ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958 not found: ID does not exist" containerID="ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958"
Oct 30 22:26:03.178932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.178842    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958"} err="failed to get container status \"ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958\": rpc error: code = NotFound desc = could not find container \"ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958\": container with ID starting with ce25125643b295413b4877cc4be4ab399f92dba84baf04b6e76153d81cf8e958 not found: ID does not exist"
Oct 30 22:26:03.340981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.340940    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"]
Oct 30 22:26:03.347937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:03.347900    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4"]
Oct 30 22:26:04.508794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:04.508755    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:04.647914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:04.647874    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7787089a-e528-4c5c-b083-dc3d94fb6db4" path="/var/lib/kubelet/pods/7787089a-e528-4c5c-b083-dc3d94fb6db4/volumes"
Oct 30 22:26:04.765575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:04.765483    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:05.010264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:05.008255    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:05.832373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:05.832339    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.069613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:06.069585    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.293802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:06.293761    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.565654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:06.565556    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.874264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:06.874192    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:07.056101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:07.056049    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"]
Oct 30 22:26:07.130953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:07.130817    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4511b0-g7npf"]
Oct 30 22:26:07.134295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:07.134267    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:26:07.138024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:07.137998    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:26:07.310476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:07.310395    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:07.558595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:07.558499    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:07.956107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:07.956053    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.344167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:08.344121    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.576637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:08.576604    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.590882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:08.590854    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"]
Oct 30 22:26:08.648727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:08.648650    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc8ee34-a224-46e7-9865-cdd4eacf5bfc" path="/var/lib/kubelet/pods/edc8ee34-a224-46e7-9865-cdd4eacf5bfc/volumes"
Oct 30 22:26:08.649004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:08.648990    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"]
Oct 30 22:26:08.696168 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:26:08.696130    5031 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-api-a-512caa-dvl5s\" is forbidden: User \"system:node:ip-10-0-141-147.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-147.us-west-1.compute.internal' and this object" podUID="0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-512caa-dvl5s"
Oct 30 22:26:09.050515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:09.050424    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:10.656160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:10.656118    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6" path="/var/lib/kubelet/pods/0a2b25bd-4dc9-4fb1-9537-e372ce7afcf6/volumes"
Oct 30 22:26:12.813863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:12.813822    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.045013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:13.044972    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.277061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:13.277020    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.525599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.525551    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"]
Oct 30 22:26:14.526492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.526470    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7787089a-e528-4c5c-b083-dc3d94fb6db4" containerName="payload-pre-processing"
Oct 30 22:26:14.526492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.526492    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7787089a-e528-4c5c-b083-dc3d94fb6db4" containerName="payload-pre-processing"
Oct 30 22:26:14.526718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.526703    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7787089a-e528-4c5c-b083-dc3d94fb6db4" containerName="payload-pre-processing"
Oct 30 22:26:14.530536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.530512    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.591069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.591038    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"]
Oct 30 22:26:14.637073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.637029    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:14.637285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.637102    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:14.637285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.637102    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:14.637285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.637201    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:26:14.687284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.687254    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dgx\" (UniqueName: \"kubernetes.io/projected/dc4b21d1-8337-4416-b9a5-d05a0173053d-kube-api-access-98dgx\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.687495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.687468    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.687547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.687517    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.714618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.714589    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.788960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.788876    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.788960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.788930    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.789140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.788988    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dgx\" (UniqueName: \"kubernetes.io/projected/dc4b21d1-8337-4416-b9a5-d05a0173053d-kube-api-access-98dgx\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.789909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.789718    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.791671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.791647    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.853101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.853069    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dgx\" (UniqueName: \"kubernetes.io/projected/dc4b21d1-8337-4416-b9a5-d05a0173053d-kube-api-access-98dgx\") pod \"maas-api-revoke-sub-4e7bbc64-8t4xg\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:14.967630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:14.967597    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.144380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.144346    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:15.193157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.193121    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.232811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.232768    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"]
Oct 30 22:26:15.236496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.236467    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce513-fh9tz"]
Oct 30 22:26:15.485420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.485327    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.544746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.544695    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"]
Oct 30 22:26:15.548294 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:26:15.548260    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4b21d1_8337_4416_b9a5_d05a0173053d.slice/crio-08f32762335c8535279c9ca7e8c587a6cec1b076da9d8dd27bca6b16dfc8adbb WatchSource:0}: Error finding container 08f32762335c8535279c9ca7e8c587a6cec1b076da9d8dd27bca6b16dfc8adbb: Status 404 returned error can't find the container with id 08f32762335c8535279c9ca7e8c587a6cec1b076da9d8dd27bca6b16dfc8adbb
Oct 30 22:26:15.718554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.718516    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.937413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:15.937387    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:16.283505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:16.283404    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg" event={"ID":"dc4b21d1-8337-4416-b9a5-d05a0173053d","Type":"ContainerStarted","Data":"1945f30cf444f0a18a934a3a462163b9b75b4fb71a118e745ad68e159c4f698a"}
Oct 30 22:26:16.283505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:16.283455    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg" event={"ID":"dc4b21d1-8337-4416-b9a5-d05a0173053d","Type":"ContainerStarted","Data":"08f32762335c8535279c9ca7e8c587a6cec1b076da9d8dd27bca6b16dfc8adbb"}
Oct 30 22:26:16.403493 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:16.403440    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg" podStartSLOduration=2.403425568 podStartE2EDuration="2.403425568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:16.402669567 +0000 UTC m=+2187.926812331" watchObservedRunningTime="2026-10-30 22:26:16.403425568 +0000 UTC m=+2187.927568331"
Oct 30 22:26:16.646360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:16.646323    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7656e4da-091b-4fdd-9b46-cda1181ecedf" path="/var/lib/kubelet/pods/7656e4da-091b-4fdd-9b46-cda1181ecedf/volumes"
Oct 30 22:26:16.799560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:16.799525    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:17.024447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:17.024346    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:17.292856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:17.292772    5031 generic.go:356] "Generic (PLEG): container finished" podID="dc4b21d1-8337-4416-b9a5-d05a0173053d" containerID="1945f30cf444f0a18a934a3a462163b9b75b4fb71a118e745ad68e159c4f698a" exitCode=0
Oct 30 22:26:17.293047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:17.292867    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg" event={"ID":"dc4b21d1-8337-4416-b9a5-d05a0173053d","Type":"ContainerDied","Data":"1945f30cf444f0a18a934a3a462163b9b75b4fb71a118e745ad68e159c4f698a"}
Oct 30 22:26:18.133271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.133236    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"]
Oct 30 22:26:18.133761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.133458    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy" containerID="cri-o://e81150e6716343faef43211b7219d6dd044cdbf72b8b1bfba65b699dc6741d00" gracePeriod=30
Oct 30 22:26:18.482525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.482494    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:18.648453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.648417    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-xks-ca\") pod \"dc4b21d1-8337-4416-b9a5-d05a0173053d\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") "
Oct 30 22:26:18.648624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.648512    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4b21d1-8337-4416-b9a5-d05a0173053d-kube-api-access-98dgx\" (UniqueName: \"kubernetes.io/projected/dc4b21d1-8337-4416-b9a5-d05a0173053d-kube-api-access-98dgx\") pod \"dc4b21d1-8337-4416-b9a5-d05a0173053d\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") "
Oct 30 22:26:18.648624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.648540    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-openshift-ca\") pod \"dc4b21d1-8337-4416-b9a5-d05a0173053d\" (UID: \"dc4b21d1-8337-4416-b9a5-d05a0173053d\") "
Oct 30 22:26:18.648998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.648974    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-openshift-ca" pod "dc4b21d1-8337-4416-b9a5-d05a0173053d" (UID: "dc4b21d1-8337-4416-b9a5-d05a0173053d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:18.650894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.650866    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-xks-ca" pod "dc4b21d1-8337-4416-b9a5-d05a0173053d" (UID: "dc4b21d1-8337-4416-b9a5-d05a0173053d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:18.651027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.650893    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4b21d1-8337-4416-b9a5-d05a0173053d-kube-api-access-98dgx" pod "dc4b21d1-8337-4416-b9a5-d05a0173053d" (UID: "dc4b21d1-8337-4416-b9a5-d05a0173053d"). InnerVolumeSpecName "kube-api-access-98dgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:18.750428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.750332    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:18.750428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.750374    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98dgx\" (UniqueName: \"kubernetes.io/projected/dc4b21d1-8337-4416-b9a5-d05a0173053d-kube-api-access-98dgx\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:18.750428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.750392    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4b21d1-8337-4416-b9a5-d05a0173053d-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:18.803021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.802980    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:18.921632 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.921597    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"]
Oct 30 22:26:18.922425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.922404    5031 cpu_manager.go:414] "containerMap: removing container" podUID="dc4b21d1-8337-4416-b9a5-d05a0173053d" containerName="revoke-keys"
Oct 30 22:26:18.922425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.922423    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4b21d1-8337-4416-b9a5-d05a0173053d" containerName="revoke-keys"
Oct 30 22:26:18.922618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.922562    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4b21d1-8337-4416-b9a5-d05a0173053d" containerName="revoke-keys"
Oct 30 22:26:18.926490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.926467    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:18.953331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.953296    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:18.953498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.953421    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:18.953576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.953560    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmph\" (UniqueName: \"kubernetes.io/projected/caddebf7-39f1-48bb-93f5-c6ef10a5059b-kube-api-access-lbmph\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:18.984260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:18.984224    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"]
Oct 30 22:26:19.029839 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.029742    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:19.054461 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.054415    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:19.054647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.054519    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmph\" (UniqueName: \"kubernetes.io/projected/caddebf7-39f1-48bb-93f5-c6ef10a5059b-kube-api-access-lbmph\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:19.054647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.054584    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:19.055132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.055104    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:19.057280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.057257    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:19.118287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.118255    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmph\" (UniqueName: \"kubernetes.io/projected/caddebf7-39f1-48bb-93f5-c6ef10a5059b-kube-api-access-lbmph\") pod \"maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:19.244767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.244732    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:19.342264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.326404    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg" event={"ID":"dc4b21d1-8337-4416-b9a5-d05a0173053d","Type":"ContainerDied","Data":"08f32762335c8535279c9ca7e8c587a6cec1b076da9d8dd27bca6b16dfc8adbb"}
Oct 30 22:26:19.342264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.326471    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f32762335c8535279c9ca7e8c587a6cec1b076da9d8dd27bca6b16dfc8adbb"
Oct 30 22:26:19.342264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.326548    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"
Oct 30 22:26:19.638475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:19.638445    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"]
Oct 30 22:26:19.638935 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:26:19.638905    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaddebf7_39f1_48bb_93f5_c6ef10a5059b.slice/crio-297b8ee68c028fc52306b49af1f8fd0518d0f340aa7dfa84cc8690dda3f437ea WatchSource:0}: Error finding container 297b8ee68c028fc52306b49af1f8fd0518d0f340aa7dfa84cc8690dda3f437ea: Status 404 returned error can't find the container with id 297b8ee68c028fc52306b49af1f8fd0518d0f340aa7dfa84cc8690dda3f437ea
Oct 30 22:26:20.335616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:20.335525    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g" event={"ID":"caddebf7-39f1-48bb-93f5-c6ef10a5059b","Type":"ContainerStarted","Data":"d552f7577545c915c52062d148eaaa160c36cf48be118db0efaa88503c95a376"}
Oct 30 22:26:20.335616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:20.335590    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g" event={"ID":"caddebf7-39f1-48bb-93f5-c6ef10a5059b","Type":"ContainerStarted","Data":"297b8ee68c028fc52306b49af1f8fd0518d0f340aa7dfa84cc8690dda3f437ea"}
Oct 30 22:26:20.463605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:20.463544    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g" podStartSLOduration=2.463529876 podStartE2EDuration="2.463529876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:20.459621948 +0000 UTC m=+2191.983764712" watchObservedRunningTime="2026-10-30 22:26:20.463529876 +0000 UTC m=+2191.987672640"
Oct 30 22:26:20.544032 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:26:20.543996    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:26:20.670081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:20.670045    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:21.346381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:21.346336    5031 generic.go:356] "Generic (PLEG): container finished" podID="caddebf7-39f1-48bb-93f5-c6ef10a5059b" containerID="d552f7577545c915c52062d148eaaa160c36cf48be118db0efaa88503c95a376" exitCode=0
Oct 30 22:26:21.346811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:21.346408    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g" event={"ID":"caddebf7-39f1-48bb-93f5-c6ef10a5059b","Type":"ContainerDied","Data":"d552f7577545c915c52062d148eaaa160c36cf48be118db0efaa88503c95a376"}
Oct 30 22:26:22.513169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.513142    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:22.599711 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.599673    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-xks-ca\") pod \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") "
Oct 30 22:26:22.599924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.599748    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-openshift-ca\") pod \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") "
Oct 30 22:26:22.599924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.599906    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caddebf7-39f1-48bb-93f5-c6ef10a5059b-kube-api-access-lbmph\" (UniqueName: \"kubernetes.io/projected/caddebf7-39f1-48bb-93f5-c6ef10a5059b-kube-api-access-lbmph\") pod \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\" (UID: \"caddebf7-39f1-48bb-93f5-c6ef10a5059b\") "
Oct 30 22:26:22.600093 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.600066    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-openshift-ca" pod "caddebf7-39f1-48bb-93f5-c6ef10a5059b" (UID: "caddebf7-39f1-48bb-93f5-c6ef10a5059b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:22.600473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.600452    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:22.602026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.601999    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-xks-ca" pod "caddebf7-39f1-48bb-93f5-c6ef10a5059b" (UID: "caddebf7-39f1-48bb-93f5-c6ef10a5059b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:22.602253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.602228    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caddebf7-39f1-48bb-93f5-c6ef10a5059b-kube-api-access-lbmph" pod "caddebf7-39f1-48bb-93f5-c6ef10a5059b" (UID: "caddebf7-39f1-48bb-93f5-c6ef10a5059b"). InnerVolumeSpecName "kube-api-access-lbmph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:22.701357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.701295    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddebf7-39f1-48bb-93f5-c6ef10a5059b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:22.701357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:22.701329    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmph\" (UniqueName: \"kubernetes.io/projected/caddebf7-39f1-48bb-93f5-c6ef10a5059b-kube-api-access-lbmph\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:22.731646 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:26:22.731610    5031 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaddebf7_39f1_48bb_93f5_c6ef10a5059b.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:23.045828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.045725    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.274697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.264996    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.370689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.370646    5031 generic.go:356] "Generic (PLEG): container finished" podID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerID="e81150e6716343faef43211b7219d6dd044cdbf72b8b1bfba65b699dc6741d00" exitCode=0
Oct 30 22:26:23.370981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.370954    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" event={"ID":"d35b2afc-b437-4e81-ad4d-4badc5ab4179","Type":"ContainerDied","Data":"e81150e6716343faef43211b7219d6dd044cdbf72b8b1bfba65b699dc6741d00"}
Oct 30 22:26:23.371072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.371033    5031 scope.go:122] "RemoveContainer" containerID="98155c44653c6da379b3365178c291c2e13c58da9fa3e50c0644465eaf6dac37"
Oct 30 22:26:23.374673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.374639    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g" event={"ID":"caddebf7-39f1-48bb-93f5-c6ef10a5059b","Type":"ContainerDied","Data":"297b8ee68c028fc52306b49af1f8fd0518d0f340aa7dfa84cc8690dda3f437ea"}
Oct 30 22:26:23.374844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.374698    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"
Oct 30 22:26:23.374844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.374701    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="297b8ee68c028fc52306b49af1f8fd0518d0f340aa7dfa84cc8690dda3f437ea"
Oct 30 22:26:23.515309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.515272    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.518141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.518114    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:26:23.627890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.627858    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-certs\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.627934    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-podinfo\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.627966    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-kube-api-access-gnm9v\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-kube-api-access-gnm9v\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628017    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-token\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-token\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628190    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-certs" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:23.628276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628193    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-socket\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628299    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-data\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628353    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istiod-ca-cert\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628451    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-socket" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:23.628512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628504    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-envoy\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628548    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-data" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:23.628628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628567    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-credential-socket\") pod \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\" (UID: \"d35b2afc-b437-4e81-ad4d-4badc5ab4179\") "
Oct 30 22:26:23.628967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.628836    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istiod-ca-cert" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:23.629168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.629147    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.629273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.629177    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.629273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.629193    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.629273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.629227    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.629273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.629150    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-credential-socket" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:23.630646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.630622    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-token" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:23.630797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.630783    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-kube-api-access-gnm9v" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "kube-api-access-gnm9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:23.631185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.631163    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-envoy" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:23.631185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.631166    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-podinfo" pod "d35b2afc-b437-4e81-ad4d-4badc5ab4179" (UID: "d35b2afc-b437-4e81-ad4d-4badc5ab4179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:23.730845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.730798    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.730845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.730836    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35b2afc-b437-4e81-ad4d-4badc5ab4179-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.730845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.730851    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.731169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.730866    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnm9v\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-kube-api-access-gnm9v\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.731169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:23.730879    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35b2afc-b437-4e81-ad4d-4badc5ab4179-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:24.384834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:24.384803    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"
Oct 30 22:26:24.385047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:24.384833    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf" event={"ID":"d35b2afc-b437-4e81-ad4d-4badc5ab4179","Type":"ContainerDied","Data":"de2bfba880106b1b8820567cd62855d805a4b089257de4ecd4af4753f4125121"}
Oct 30 22:26:24.385047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:24.384892    5031 scope.go:122] "RemoveContainer" containerID="e81150e6716343faef43211b7219d6dd044cdbf72b8b1bfba65b699dc6741d00"
Oct 30 22:26:24.615897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:24.615860    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"]
Oct 30 22:26:24.664171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:24.664094    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf"]
Oct 30 22:26:25.330728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:25.330688    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:26.140319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:26.140290    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:26.649967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:26.649920    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" path="/var/lib/kubelet/pods/d35b2afc-b437-4e81-ad4d-4badc5ab4179/volumes"
Oct 30 22:26:27.714578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:27.714485    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:28.078636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:28.078553    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:29.763522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:29.763490    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:30.028791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:30.028715    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:31.350006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:31.349962    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:34.072702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:34.072664    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:35.937494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:35.937459    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.612299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:36.612259    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:39.978086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:39.978049    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"]
Oct 30 22:26:39.978633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:39.978289    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy" containerID="cri-o://c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6" gracePeriod=30
Oct 30 22:26:40.987602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:40.987519    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:41.253056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:41.252955    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:41.541820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:41.541731    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.610411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:42.610373    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.834182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:42.834146    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:43.055492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:43.055390    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"]
Oct 30 22:26:43.059185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:43.059138    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:43.106690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:43.106663    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c3d917-72zdl"]
Oct 30 22:26:43.595544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:43.595513    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:44.646761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:44.646721    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82c5bca-f443-43da-9df4-030c1b7aa6bb" path="/var/lib/kubelet/pods/c82c5bca-f443-43da-9df4-030c1b7aa6bb/volumes"
Oct 30 22:26:45.246659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.246632    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:26:45.338041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.337944    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-kube-api-access-2nh5k\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-kube-api-access-2nh5k\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.338476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.338452    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-socket\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.338628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.338615    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-podinfo\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.338746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.338734    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-credential-socket\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.338847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.338833    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-certs\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.338986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.338973    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istiod-ca-cert\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.339292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.339273    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-token\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-token\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.339449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.339436    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-envoy\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.339597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.339584    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-data\") pod \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\" (UID: \"82f61a9c-f88c-4bad-baa0-b630b6e6184e\") "
Oct 30 22:26:45.340471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.340447    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-data" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:45.341358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.341167    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-certs" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:45.341780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.341502    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-socket" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:45.342682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.342660    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istiod-ca-cert" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:45.343061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.343041    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-credential-socket" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:45.352702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.352669    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-kube-api-access-2nh5k" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "kube-api-access-2nh5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:45.355725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.355695    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-token" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:45.356226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.356161    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-podinfo" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:45.356765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.356731    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-envoy" pod "82f61a9c-f88c-4bad-baa0-b630b6e6184e" (UID: "82f61a9c-f88c-4bad-baa0-b630b6e6184e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:45.446786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446748    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.446786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446788    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.447478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446808    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.447478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446823    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nh5k\" (UniqueName: \"kubernetes.io/projected/82f61a9c-f88c-4bad-baa0-b630b6e6184e-kube-api-access-2nh5k\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.447478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446840    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.447478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446857    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.447478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446872    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.447478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446886    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f61a9c-f88c-4bad-baa0-b630b6e6184e-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.447478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.446899    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f61a9c-f88c-4bad-baa0-b630b6e6184e-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.583618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.583584    5031 generic.go:356] "Generic (PLEG): container finished" podID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerID="c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6" exitCode=0
Oct 30 22:26:45.583832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.583640    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"
Oct 30 22:26:45.583832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.583664    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" event={"ID":"82f61a9c-f88c-4bad-baa0-b630b6e6184e","Type":"ContainerDied","Data":"c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6"}
Oct 30 22:26:45.583832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.583722    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px" event={"ID":"82f61a9c-f88c-4bad-baa0-b630b6e6184e","Type":"ContainerDied","Data":"f599c35cf6585c6379fcd1311f1a47cb333ef0fec0622809f791b47160c028da"}
Oct 30 22:26:45.583832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.583749    5031 scope.go:122] "RemoveContainer" containerID="c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6"
Oct 30 22:26:45.596354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.596336    5031 scope.go:122] "RemoveContainer" containerID="c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6"
Oct 30 22:26:45.596628 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:26:45.596607    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6\": container with ID starting with c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6 not found: ID does not exist" containerID="c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6"
Oct 30 22:26:45.596704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.596642    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6"} err="failed to get container status \"c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6\": rpc error: code = NotFound desc = could not find container \"c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6\": container with ID starting with c540dbf633403f94e5a29a0aad553776272f5da6d17e56997f6985dd1f1063d6 not found: ID does not exist"
Oct 30 22:26:45.755595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.755561    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"]
Oct 30 22:26:45.761271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:45.761228    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px"]
Oct 30 22:26:46.646879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:46.646843    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" path="/var/lib/kubelet/pods/82f61a9c-f88c-4bad-baa0-b630b6e6184e/volumes"
Oct 30 22:26:48.178812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:48.178778    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.408751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:48.408714    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.644777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:48.644740    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:49.539840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:49.539809    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:50.543809 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:26:50.543763    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:26:51.225176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:51.225144    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:51.459028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:51.458993    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:51.520443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:51.520412    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"]
Oct 30 22:26:51.520752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:51.520692    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy" containerID="cri-o://238a5ecfc07f41066ffbf6085df1737a707abeb9460391f2662e4bf87576a543" gracePeriod=30
Oct 30 22:26:51.683817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:51.683780    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:52.186171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.186134    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:52.681939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.681902    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"]
Oct 30 22:26:52.682553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682539    5031 cpu_manager.go:414] "containerMap: removing container" podUID="caddebf7-39f1-48bb-93f5-c6ef10a5059b" containerName="revoke-keys"
Oct 30 22:26:52.682645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682554    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caddebf7-39f1-48bb-93f5-c6ef10a5059b" containerName="revoke-keys"
Oct 30 22:26:52.682645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682570    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy"
Oct 30 22:26:52.682645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682575    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy"
Oct 30 22:26:52.682645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682601    5031 cpu_manager.go:414] "containerMap: removing container" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy"
Oct 30 22:26:52.682645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682606    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy"
Oct 30 22:26:52.682814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682702    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy"
Oct 30 22:26:52.682814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682714    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f61a9c-f88c-4bad-baa0-b630b6e6184e" containerName="istio-proxy"
Oct 30 22:26:52.682814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.682724    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="caddebf7-39f1-48bb-93f5-c6ef10a5059b" containerName="revoke-keys"
Oct 30 22:26:52.688036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.688018    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.754302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.754267    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"]
Oct 30 22:26:52.763743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.763712    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.763940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.763773    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9wz\" (UniqueName: \"kubernetes.io/projected/1c60101e-ad73-4939-b119-60f08ab0e91b-kube-api-access-jk9wz\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.763940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.763864    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.800177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.800137    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:52.800177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.800165    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:52.800177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.800182    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:52.800469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.800259    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:26:52.865180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.865147    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.865387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.865201    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9wz\" (UniqueName: \"kubernetes.io/projected/1c60101e-ad73-4939-b119-60f08ab0e91b-kube-api-access-jk9wz\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.865387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.865250    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.866032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.865952    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.867870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.867841    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:52.930643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:52.930613    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9wz\" (UniqueName: \"kubernetes.io/projected/1c60101e-ad73-4939-b119-60f08ab0e91b-kube-api-access-jk9wz\") pod \"maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:53.002448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:53.002347    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:53.388017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:53.387981    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"]
Oct 30 22:26:53.390175 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:26:53.390147    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c60101e_ad73_4939_b119_60f08ab0e91b.slice/crio-095d17db6186678ef9e7d62a48ccefa5693f7687fe2c86fb2547cf0de13e5ace WatchSource:0}: Error finding container 095d17db6186678ef9e7d62a48ccefa5693f7687fe2c86fb2547cf0de13e5ace: Status 404 returned error can't find the container with id 095d17db6186678ef9e7d62a48ccefa5693f7687fe2c86fb2547cf0de13e5ace
Oct 30 22:26:53.575569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:53.575536    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.673680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:53.673598    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw" event={"ID":"1c60101e-ad73-4939-b119-60f08ab0e91b","Type":"ContainerStarted","Data":"095d17db6186678ef9e7d62a48ccefa5693f7687fe2c86fb2547cf0de13e5ace"}
Oct 30 22:26:54.683246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:54.683182    5031 generic.go:356] "Generic (PLEG): container finished" podID="1c60101e-ad73-4939-b119-60f08ab0e91b" containerID="6eebf0aa062c762cbe3b7953b22bfd25e983612441272e72232eb9a4be1795d8" exitCode=0
Oct 30 22:26:54.683695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:54.683266    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw" event={"ID":"1c60101e-ad73-4939-b119-60f08ab0e91b","Type":"ContainerDied","Data":"6eebf0aa062c762cbe3b7953b22bfd25e983612441272e72232eb9a4be1795d8"}
Oct 30 22:26:55.115399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.115363    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"]
Oct 30 22:26:55.116040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.116022    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy"
Oct 30 22:26:55.116093 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.116042    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy"
Oct 30 22:26:55.116147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.116138    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35b2afc-b437-4e81-ad4d-4badc5ab4179" containerName="istio-proxy"
Oct 30 22:26:55.120154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.120127    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.177567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.177536    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-422c1029-openshift-default-dockercfg-f8xcc\""
Oct 30 22:26:55.222074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.221799    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"]
Oct 30 22:26:55.297822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.297789    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.297840    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvvm\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-kube-api-access-fvvvm\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.298010    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.298054    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997a6178-1375-4f98-b1b0-4f4939da3fe9-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.298149    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.298246    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.298297    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.298380    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.298471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.298434    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.399902 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.399807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.399911    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.399952    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvvm\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-kube-api-access-fvvvm\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.400045    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.400072    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997a6178-1375-4f98-b1b0-4f4939da3fe9-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.400101    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.400160    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.400244    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.400387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.400297    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.401406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.401373    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.401558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.401510    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.401558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.401522    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.401697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.401655    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.402080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.402052    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997a6178-1375-4f98-b1b0-4f4939da3fe9-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.402994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.402973    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.403891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.403870    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.464169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.464134    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.464347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.464258    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvvm\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-kube-api-access-fvvvm\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.734509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.734475    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:55.862835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:55.862805    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:56.007734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.007641    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-xks-ca\") pod \"1c60101e-ad73-4939-b119-60f08ab0e91b\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") "
Oct 30 22:26:56.007906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.007793    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-openshift-ca\") pod \"1c60101e-ad73-4939-b119-60f08ab0e91b\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") "
Oct 30 22:26:56.007906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.007897    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c60101e-ad73-4939-b119-60f08ab0e91b-kube-api-access-jk9wz\" (UniqueName: \"kubernetes.io/projected/1c60101e-ad73-4939-b119-60f08ab0e91b-kube-api-access-jk9wz\") pod \"1c60101e-ad73-4939-b119-60f08ab0e91b\" (UID: \"1c60101e-ad73-4939-b119-60f08ab0e91b\") "
Oct 30 22:26:56.008172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.008142    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-openshift-ca" pod "1c60101e-ad73-4939-b119-60f08ab0e91b" (UID: "1c60101e-ad73-4939-b119-60f08ab0e91b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:56.008557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.008528    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.010176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.010144    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-xks-ca" pod "1c60101e-ad73-4939-b119-60f08ab0e91b" (UID: "1c60101e-ad73-4939-b119-60f08ab0e91b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:56.010376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.010323    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c60101e-ad73-4939-b119-60f08ab0e91b-kube-api-access-jk9wz" pod "1c60101e-ad73-4939-b119-60f08ab0e91b" (UID: "1c60101e-ad73-4939-b119-60f08ab0e91b"). InnerVolumeSpecName "kube-api-access-jk9wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:56.109606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.109571    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c60101e-ad73-4939-b119-60f08ab0e91b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.109606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.109601    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk9wz\" (UniqueName: \"kubernetes.io/projected/1c60101e-ad73-4939-b119-60f08ab0e91b-kube-api-access-jk9wz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.203036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.203003    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"]
Oct 30 22:26:56.203293 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:26:56.203271    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997a6178_1375_4f98_b1b0_4f4939da3fe9.slice/crio-ea4a7140535208fa5cbd693f3fb8526bffeb368173b290b6f37bd6c4bf767a9e WatchSource:0}: Error finding container ea4a7140535208fa5cbd693f3fb8526bffeb368173b290b6f37bd6c4bf767a9e: Status 404 returned error can't find the container with id ea4a7140535208fa5cbd693f3fb8526bffeb368173b290b6f37bd6c4bf767a9e
Oct 30 22:26:56.206434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.205866    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:56.206434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.205950    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:56.206434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.206005    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:56.702441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.702407    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"
Oct 30 22:26:56.702441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.702413    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw" event={"ID":"1c60101e-ad73-4939-b119-60f08ab0e91b","Type":"ContainerDied","Data":"095d17db6186678ef9e7d62a48ccefa5693f7687fe2c86fb2547cf0de13e5ace"}
Oct 30 22:26:56.702693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.702480    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="095d17db6186678ef9e7d62a48ccefa5693f7687fe2c86fb2547cf0de13e5ace"
Oct 30 22:26:56.704675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.704625    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" event={"ID":"997a6178-1375-4f98-b1b0-4f4939da3fe9","Type":"ContainerStarted","Data":"40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f"}
Oct 30 22:26:56.704865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.704696    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" event={"ID":"997a6178-1375-4f98-b1b0-4f4939da3fe9","Type":"ContainerStarted","Data":"ea4a7140535208fa5cbd693f3fb8526bffeb368173b290b6f37bd6c4bf767a9e"}
Oct 30 22:26:56.707276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.707198    5031 generic.go:356] "Generic (PLEG): container finished" podID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerID="238a5ecfc07f41066ffbf6085df1737a707abeb9460391f2662e4bf87576a543" exitCode=0
Oct 30 22:26:56.707400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.707256    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" event={"ID":"609f40a6-6a81-4eaa-b191-e414cf8bc4cf","Type":"ContainerDied","Data":"238a5ecfc07f41066ffbf6085df1737a707abeb9460391f2662e4bf87576a543"}
Oct 30 22:26:56.707400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.707380    5031 scope.go:122] "RemoveContainer" containerID="0892340848cbd133c527e638e31ba9597ece28bf3c9c7effc1adc68505c5b06f"
Oct 30 22:26:56.738416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.738328    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:26:56.848597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.848531    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" podStartSLOduration=1.8485118379999999 podStartE2EDuration="1.848511838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:56.845254403 +0000 UTC m=+2228.369397291" watchObservedRunningTime="2026-10-30 22:26:56.848511838 +0000 UTC m=+2228.372654605"
Oct 30 22:26:56.874192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:56.874167    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:26:57.036541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036506    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-socket\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.036541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036546    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-podinfo\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.036832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036613    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-data\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.036832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036660    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-kube-api-access-mkjgg\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-kube-api-access-mkjgg\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.036832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036788    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-certs\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.036832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036820    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-socket" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:57.036832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036828    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-credential-socket\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.037089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036926    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-data" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:57.037089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036945    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-token\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-token\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.037089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.036981    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istiod-ca-cert\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.037089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037015    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-envoy\") pod \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\" (UID: \"609f40a6-6a81-4eaa-b191-e414cf8bc4cf\") "
Oct 30 22:26:57.037347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037176    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-credential-socket" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:57.037434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037411    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istiod-ca-cert" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:57.037554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037533    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-certs" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:57.037710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037692    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.037790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037717    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.037790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037732    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.037790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037745    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.037790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.037761    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.039494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.039466    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-token" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:57.039651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.039493    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-envoy" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:57.039731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.039706    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-kube-api-access-mkjgg" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "kube-api-access-mkjgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:57.039796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.039769    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-podinfo" pod "609f40a6-6a81-4eaa-b191-e414cf8bc4cf" (UID: "609f40a6-6a81-4eaa-b191-e414cf8bc4cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:57.138571 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.138541    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.138571 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.138569    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.138571 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.138578    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.138856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.138589    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkjgg\" (UniqueName: \"kubernetes.io/projected/609f40a6-6a81-4eaa-b191-e414cf8bc4cf-kube-api-access-mkjgg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.233118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.233082    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:57.736291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.730727    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"
Oct 30 22:26:57.736291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.732548    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg" event={"ID":"609f40a6-6a81-4eaa-b191-e414cf8bc4cf","Type":"ContainerDied","Data":"f4795ca2d93fbdd268589cc8ff6035a504cd2b9117b81115547df6524f631ea4"}
Oct 30 22:26:57.736291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.732611    5031 scope.go:122] "RemoveContainer" containerID="238a5ecfc07f41066ffbf6085df1737a707abeb9460391f2662e4bf87576a543"
Oct 30 22:26:57.739666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.739638    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:26:57.740199 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.740168    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:26:57.904513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.904480    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"]
Oct 30 22:26:57.955075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.955034    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg"]
Oct 30 22:26:57.993317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:57.993226    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:58.647245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:58.647194    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" path="/var/lib/kubelet/pods/609f40a6-6a81-4eaa-b191-e414cf8bc4cf/volumes"
Oct 30 22:26:58.736270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:58.735977    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:26:58.736270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:58.736057    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:26:59.177995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.177959    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:59.735327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.735285    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:26:59.735531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.735354    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:26:59.891562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.891516    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"]
Oct 30 22:26:59.892494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.892471    5031 cpu_manager.go:414] "containerMap: removing container" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy"
Oct 30 22:26:59.893027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.892499    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy"
Oct 30 22:26:59.893027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.892576    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1c60101e-ad73-4939-b119-60f08ab0e91b" containerName="revoke-keys"
Oct 30 22:26:59.893027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.892584    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c60101e-ad73-4939-b119-60f08ab0e91b" containerName="revoke-keys"
Oct 30 22:26:59.893027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.892771    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c60101e-ad73-4939-b119-60f08ab0e91b" containerName="revoke-keys"
Oct 30 22:26:59.893027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.892787    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy"
Oct 30 22:26:59.893027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.892797    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy"
Oct 30 22:26:59.898587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.898564    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978161    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978264    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfns\" (UniqueName: \"kubernetes.io/projected/b622abc4-5b50-4f1b-84cc-30edabc33258-kube-api-access-trfns\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978322    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978351    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978392    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b622abc4-5b50-4f1b-84cc-30edabc33258-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978479    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978548    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978622    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:26:59.978737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:26:59.978666    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.080243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.080119    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.080782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.080758    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.081601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.081581    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.081728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.081716    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.081837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.081821    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.081977 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.081966    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.082063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.082052    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfns\" (UniqueName: \"kubernetes.io/projected/b622abc4-5b50-4f1b-84cc-30edabc33258-kube-api-access-trfns\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.082171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.082157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.082269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.082258    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.082369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.082359    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b622abc4-5b50-4f1b-84cc-30edabc33258-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.083087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.083070    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b622abc4-5b50-4f1b-84cc-30edabc33258-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.085366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.083534    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.085366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.083747    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.085366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.083954    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.085366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.084246    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b622abc4-5b50-4f1b-84cc-30edabc33258-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.089068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.089032    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.146145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.146071    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"]
Oct 30 22:27:00.147814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.147770    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfns\" (UniqueName: \"kubernetes.io/projected/b622abc4-5b50-4f1b-84cc-30edabc33258-kube-api-access-trfns\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.148604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.148582    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b622abc4-5b50-4f1b-84cc-30edabc33258-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nqjll\" (UID: \"b622abc4-5b50-4f1b-84cc-30edabc33258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.247311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.247269    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:00.733051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.733011    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"]
Oct 30 22:27:00.733798 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:27:00.733765    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb622abc4_5b50_4f1b_84cc_30edabc33258.slice/crio-25cfb10ef4ecc90ddc41c561859c3418b71553062b59d350a37cae918f66ac19 WatchSource:0}: Error finding container 25cfb10ef4ecc90ddc41c561859c3418b71553062b59d350a37cae918f66ac19: Status 404 returned error can't find the container with id 25cfb10ef4ecc90ddc41c561859c3418b71553062b59d350a37cae918f66ac19
Oct 30 22:27:00.734924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.734897    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:27:00.735023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.734968    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:27:00.736723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.736689    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.736823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.736763    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.736823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.736796    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:00.762986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:00.762958    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" event={"ID":"b622abc4-5b50-4f1b-84cc-30edabc33258","Type":"ContainerStarted","Data":"25cfb10ef4ecc90ddc41c561859c3418b71553062b59d350a37cae918f66ac19"}
Oct 30 22:27:01.739148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:01.738626    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:27:01.739148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:01.738693    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:27:01.772766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:01.772729    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" event={"ID":"b622abc4-5b50-4f1b-84cc-30edabc33258","Type":"ContainerStarted","Data":"61ce4c0a2f5891ea29e5b36dc97bc50c619f7b70fbfa5440e5fd37162f29ee0d"}
Oct 30 22:27:01.953269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:01.953138    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podStartSLOduration=2.953115767 podStartE2EDuration="2.953115767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:01.922760749 +0000 UTC m=+2233.446903516" watchObservedRunningTime="2026-10-30 22:27:01.953115767 +0000 UTC m=+2233.477258532"
Oct 30 22:27:02.251236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:02.251127    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:02.251477 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:02.251403    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:27:02.251570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:02.251480    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 22:27:02.980685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:02.980647    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"]
Oct 30 22:27:03.096171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.096128    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"]
Oct 30 22:27:03.251665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.251569    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:27:03.251980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.251942    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 22:27:03.444238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.411877    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:27:03.460241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.445828    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:27:03.461921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.461767    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:03.473247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.465443    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:27:03.682634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:03.682589    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:04.247708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:04.247665    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:27:04.248148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:04.247721    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 22:27:05.245476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:05.245433    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:05.248682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:05.248168    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:27:05.248682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:05.248262    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 22:27:06.247917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:06.247878    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:27:06.248146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:06.247970    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 22:27:07.248584 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:07.248544    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:27:07.249118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:07.248610    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 22:27:07.981414 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:07.981354    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" podUID="d1820885-4e6b-40aa-9c73-f64219b771dd" containerName="payload-processing" containerID="cri-o://a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15" gracePeriod=25
Oct 30 22:27:08.097556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.097517    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" podUID="129685bc-38ce-4438-974d-bb2ea0c1a743" containerName="payload-pre-processing" containerID="cri-o://aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286" gracePeriod=25
Oct 30 22:27:08.247810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.247708    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:27:08.247810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.247773    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Oct 30 22:27:08.793316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.793271    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"]
Oct 30 22:27:08.793901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.793884    5031 cpu_manager.go:414] "containerMap: removing container" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy"
Oct 30 22:27:08.793901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.793900    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609f40a6-6a81-4eaa-b191-e414cf8bc4cf" containerName="istio-proxy"
Oct 30 22:27:08.797882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.797858    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.861735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.861693    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"]
Oct 30 22:27:08.877383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.877343    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:08.877560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.877385    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:08.893343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.893303    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpq7w\" (UniqueName: \"kubernetes.io/projected/22f2b52d-1411-4d94-b209-73a3ba685c60-kube-api-access-jpq7w\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.893523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.893508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.893605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.893562    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.903223 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.903185    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:08.903366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.903268    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:27:08.994649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.994612    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.994859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.994684    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.994859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.994734    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpq7w\" (UniqueName: \"kubernetes.io/projected/22f2b52d-1411-4d94-b209-73a3ba685c60-kube-api-access-jpq7w\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.995323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.995297    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:08.997378 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:08.997356    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:09.060047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:09.059969    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpq7w\" (UniqueName: \"kubernetes.io/projected/22f2b52d-1411-4d94-b209-73a3ba685c60-kube-api-access-jpq7w\") pod \"maas-api-revoke-keys-e2e-mt-422c1029-njtgv\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:09.115563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:09.115518    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:09.527884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:09.527852    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"]
Oct 30 22:27:09.867273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:09.867239    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv" event={"ID":"22f2b52d-1411-4d94-b209-73a3ba685c60","Type":"ContainerStarted","Data":"bd9c068033ad7400996020817a29b0db154be93fdad7b931b85f341049ce64b2"}
Oct 30 22:27:09.974083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:09.974052    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:09.974857 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:09.974824    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:09.975782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:09.975759    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll"
Oct 30 22:27:10.204011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.203918    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:10.263878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.263841    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"]
Oct 30 22:27:10.270107 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.270081    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.288982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.288946    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"]
Oct 30 22:27:10.326543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.326272    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.327722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.327366    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmrxj\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-kube-api-access-rmrxj\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.327722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.327583    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.328292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.328071    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.329113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.328516    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.329113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.328787    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd67607-6d80-46b0-b543-e2140d73053d-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.329113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.328965    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.335056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.335017    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd67607-6d80-46b0-b543-e2140d73053d-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.335523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.335468    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.343864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.340647    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.343864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.343251    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"]
Oct 30 22:27:10.371957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.371927    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"]
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442258    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442347    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442377    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecde0778-eb17-47cf-8417-731378f39138-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442410    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442451    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmrxj\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-kube-api-access-rmrxj\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442500    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442526    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442600    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442640    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742kw\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-kube-api-access-742kw\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442667    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442704    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd67607-6d80-46b0-b543-e2140d73053d-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442733    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecde0778-eb17-47cf-8417-731378f39138-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442766    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442800    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442822    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd67607-6d80-46b0-b543-e2140d73053d-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442849    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442897    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.442935    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.445233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.443803    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.462297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.446652    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.462297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.447306    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd67607-6d80-46b0-b543-e2140d73053d-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.462297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.447574    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.462297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.452931    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.471296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.468463    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.479234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.476304    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd67607-6d80-46b0-b543-e2140d73053d-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.507403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.507368    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.507898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.507865    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmrxj\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-kube-api-access-rmrxj\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-zct7m\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.544341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544310    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544381    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544466    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544529    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544556    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecde0778-eb17-47cf-8417-731378f39138-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544593    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544726    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544773    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742kw\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-kube-api-access-742kw\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.544783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544778    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.545026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544838    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecde0778-eb17-47cf-8417-731378f39138-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.545026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.544849    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.545188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.545146    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.545188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.545171    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.545550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.545525    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecde0778-eb17-47cf-8417-731378f39138-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.547324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.547304    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.547554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.547536    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecde0778-eb17-47cf-8417-731378f39138-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.614908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.614877    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-742kw\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-kube-api-access-742kw\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.615239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.615191    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-74nsh\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.627515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.627494    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:10.662412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.662154    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:10.888428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.886912    5031 generic.go:356] "Generic (PLEG): container finished" podID="22f2b52d-1411-4d94-b209-73a3ba685c60" containerID="bb282985019d0b0b6bb07b7698b3439d50494539c7e12da8e7c44cecefacebe7" exitCode=0
Oct 30 22:27:10.888428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.888359    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv" event={"ID":"22f2b52d-1411-4d94-b209-73a3ba685c60","Type":"ContainerDied","Data":"bb282985019d0b0b6bb07b7698b3439d50494539c7e12da8e7c44cecefacebe7"}
Oct 30 22:27:10.983960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:10.983925    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:27:11.094969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.083441    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"]
Oct 30 22:27:11.125027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.124981    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.125311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.125285    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.125464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.125445    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.188022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.187983    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"]
Oct 30 22:27:11.200572 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:27:11.194886    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecde0778_eb17_47cf_8417_731378f39138.slice/crio-6e47236323221827adbcaed2de367c5e4974853b407ec5154be9179dc4cdbf7e WatchSource:0}: Error finding container 6e47236323221827adbcaed2de367c5e4974853b407ec5154be9179dc4cdbf7e: Status 404 returned error can't find the container with id 6e47236323221827adbcaed2de367c5e4974853b407ec5154be9179dc4cdbf7e
Oct 30 22:27:11.200572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.199134    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.200572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.199244    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.200572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.199290    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.900037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.899999    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" event={"ID":"ecde0778-eb17-47cf-8417-731378f39138","Type":"ContainerStarted","Data":"ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963"}
Oct 30 22:27:11.900550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.900070    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" event={"ID":"ecde0778-eb17-47cf-8417-731378f39138","Type":"ContainerStarted","Data":"6e47236323221827adbcaed2de367c5e4974853b407ec5154be9179dc4cdbf7e"}
Oct 30 22:27:11.902154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.902127    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" event={"ID":"efd67607-6d80-46b0-b543-e2140d73053d","Type":"ContainerStarted","Data":"57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176"}
Oct 30 22:27:11.902293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.902171    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" event={"ID":"efd67607-6d80-46b0-b543-e2140d73053d","Type":"ContainerStarted","Data":"8303598aab5c30d45c41b206ee06e15f890c6d0b828a1803ea17f5d16ff9690f"}
Oct 30 22:27:11.970325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:11.970123    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:12.207609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.206594    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podStartSLOduration=2.206573883 podStartE2EDuration="2.206573883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:12.198165232 +0000 UTC m=+2243.722308016" watchObservedRunningTime="2026-10-30 22:27:12.206573883 +0000 UTC m=+2243.730716648"
Oct 30 22:27:12.217243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.216540    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podStartSLOduration=2.216525689 podStartE2EDuration="2.216525689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:12.070422009 +0000 UTC m=+2243.594564799" watchObservedRunningTime="2026-10-30 22:27:12.216525689 +0000 UTC m=+2243.740668454"
Oct 30 22:27:12.286975 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.286578    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:12.349347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.349297    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:12.387132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.387076    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-openshift-ca\") pod \"22f2b52d-1411-4d94-b209-73a3ba685c60\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") "
Oct 30 22:27:12.387376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.387170    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-xks-ca\") pod \"22f2b52d-1411-4d94-b209-73a3ba685c60\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") "
Oct 30 22:27:12.387376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.387261    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f2b52d-1411-4d94-b209-73a3ba685c60-kube-api-access-jpq7w\" (UniqueName: \"kubernetes.io/projected/22f2b52d-1411-4d94-b209-73a3ba685c60-kube-api-access-jpq7w\") pod \"22f2b52d-1411-4d94-b209-73a3ba685c60\" (UID: \"22f2b52d-1411-4d94-b209-73a3ba685c60\") "
Oct 30 22:27:12.388554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.388522    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-openshift-ca" pod "22f2b52d-1411-4d94-b209-73a3ba685c60" (UID: "22f2b52d-1411-4d94-b209-73a3ba685c60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:12.391042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.391014    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-xks-ca" pod "22f2b52d-1411-4d94-b209-73a3ba685c60" (UID: "22f2b52d-1411-4d94-b209-73a3ba685c60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:12.392635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.392613    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f2b52d-1411-4d94-b209-73a3ba685c60-kube-api-access-jpq7w" pod "22f2b52d-1411-4d94-b209-73a3ba685c60" (UID: "22f2b52d-1411-4d94-b209-73a3ba685c60"). InnerVolumeSpecName "kube-api-access-jpq7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.497368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.497271    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"]
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.619735    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.619795    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f2b52d-1411-4d94-b209-73a3ba685c60-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.619819    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpq7w\" (UniqueName: \"kubernetes.io/projected/22f2b52d-1411-4d94-b209-73a3ba685c60-kube-api-access-jpq7w\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.631070    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.681808    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.681962    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.682017    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.686582    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:12.686803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.686639    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:12.703990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.703955    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff338b-ncd8g"]
Oct 30 22:27:12.704273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:12.704255    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:13.319339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.306086    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.643235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.636813    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:13.643235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.636879    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:13.663935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.663900    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:13.664248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.664193    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:13.736986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.736948    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"
Oct 30 22:27:13.761449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.737590    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv" event={"ID":"22f2b52d-1411-4d94-b209-73a3ba685c60","Type":"ContainerDied","Data":"bd9c068033ad7400996020817a29b0db154be93fdad7b931b85f341049ce64b2"}
Oct 30 22:27:13.761802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.761779    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9c068033ad7400996020817a29b0db154be93fdad7b931b85f341049ce64b2"
Oct 30 22:27:13.876260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:13.865301    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:27:14.150799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.150749    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:27:14.152830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.151105    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:14.197247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.194326    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" podUID="129685bc-38ce-4438-974d-bb2ea0c1a743" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:14.197556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.197523    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" podUID="d1820885-4e6b-40aa-9c73-f64219b771dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:14.313237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.312310    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:14.313476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.313272    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:27:14.410297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.410112    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:14.638238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.636659    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:14.638238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.636728    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:14.665304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.663680    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:14.665304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.663744    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:14.700237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.693991    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caddebf7-39f1-48bb-93f5-c6ef10a5059b" path="/var/lib/kubelet/pods/caddebf7-39f1-48bb-93f5-c6ef10a5059b/volumes"
Oct 30 22:27:14.868237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.867347    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:14.868237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:14.867762    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:27:15.290238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.176327    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:15.290238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.177294    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:27:15.290238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.177294    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:15.290238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.217371    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:27:15.400252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.375258    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:27:15.400252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.375827    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:27:15.477794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.405919    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:27:15.477794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.406350    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:27:15.771240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.736600    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:15.771240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.736666    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:15.771240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.738270    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:15.771240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.738322    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:15.870237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:15.869293    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:27:16.195236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.181398    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:16.195236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.183134    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:16.394265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.377370    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:27:16.460344 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.419508    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:27:16.654859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.654342    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:16.654859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.654405    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:16.687604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.687480    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:16.687604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.687539    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:16.882238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.874012    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:16.921236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:16.914148    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:17.160844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.160740    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:17.182354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.182301    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:17.183732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.183701    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:17.411298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.410107    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:17.436235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.433305    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:27:17.543238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.542860    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:17.632237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.631658    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:17.632237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.631742    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:17.678169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.678070    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:17.678793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.678762    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:17.929295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.928637    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:27:17.941235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.938441    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:17.971232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:17.950309    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:27:18.121202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.121076    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:18.191342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.182804    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:18.213254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.191849    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Oct 30 22:27:18.559296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.543420    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded"
Oct 30 22:27:18.763276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.683326    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:18.763276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.683427    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:18.763276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.684015    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:18.763276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.684063    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:18.826684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.826577    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 22:27:18.981340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.981308    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:27:19.005249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:18.993890    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:27:19.018351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.005801    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:19.219409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.219357    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:19.316249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.281071    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:27:19.629238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.604539    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:19.732772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.732717    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:19.733103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.733056    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:19.733785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.733760    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:27:19.733969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.733949    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 22:27:19.817615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.799933    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:19.920293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:19.913511    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:27:20.018460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.018415    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:20.056905 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.055492    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Oct 30 22:27:20.073432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.071016    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:20.102236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.100796    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:27:20.209474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.209383    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:27:20.282902 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.282873    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:27:20.617245 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:27:20.606364    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:27:20.637237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.630759    5031 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 22:27:20.637237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.630823    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 22:27:20.837236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.836546    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:20.838609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.838156    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:20.857244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:20.856781    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:27:21.407476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:21.407424    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:21.696275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:21.694634    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:21.712359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:21.712326    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:21.712933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:21.712876    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:21.713907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:21.713886    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:27:22.081662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:22.081577    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:22.380569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:22.380529    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:22.913692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:22.913654    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"]
Oct 30 22:27:22.914688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:22.914664    5031 cpu_manager.go:414] "containerMap: removing container" podUID="22f2b52d-1411-4d94-b209-73a3ba685c60" containerName="revoke-keys"
Oct 30 22:27:22.914792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:22.914692    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f2b52d-1411-4d94-b209-73a3ba685c60" containerName="revoke-keys"
Oct 30 22:27:22.914969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:22.914954    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f2b52d-1411-4d94-b209-73a3ba685c60" containerName="revoke-keys"
Oct 30 22:27:22.920519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:22.920491    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.006298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.006259    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"]
Oct 30 22:27:23.016303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.016270    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5xp\" (UniqueName: \"kubernetes.io/projected/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-kube-api-access-vd5xp\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.016529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.016448    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.016592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.016548    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.024565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.024179    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:23.024565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.024362    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:27:23.024565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.024450    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:23.024565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.024527    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:23.118109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.118056    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.118655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.118291    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5xp\" (UniqueName: \"kubernetes.io/projected/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-kube-api-access-vd5xp\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.118655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.118441    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.119271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.119238    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.121258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.121183    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.183628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.183544    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5xp\" (UniqueName: \"kubernetes.io/projected/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-kube-api-access-vd5xp\") pod \"maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.241864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.241546    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:23.268904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.268775    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:23.662281 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:27:23.661816    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2f96aa_e6c6_4c4a_8167_e2e6f4f21606.slice/crio-ad38b5e0c8600442a56b5f6c6fe47eb1f14763129db0a56512034d1374cd9e83 WatchSource:0}: Error finding container ad38b5e0c8600442a56b5f6c6fe47eb1f14763129db0a56512034d1374cd9e83: Status 404 returned error can't find the container with id ad38b5e0c8600442a56b5f6c6fe47eb1f14763129db0a56512034d1374cd9e83
Oct 30 22:27:23.662533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:23.662508    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"]
Oct 30 22:27:24.149324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:24.149274    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" podUID="129685bc-38ce-4438-974d-bb2ea0c1a743" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:24.149710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:24.149345    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" podUID="d1820885-4e6b-40aa-9c73-f64219b771dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:24.634504 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:24.634473    5031 generic.go:356] "Generic (PLEG): container finished" podID="3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" containerID="bcfc36b25e84bb7ffa7baa8ad53286c39eda4384b201e85c6e1834170da140a7" exitCode=0
Oct 30 22:27:24.634654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:24.634559    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg" event={"ID":"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606","Type":"ContainerDied","Data":"bcfc36b25e84bb7ffa7baa8ad53286c39eda4384b201e85c6e1834170da140a7"}
Oct 30 22:27:24.634654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:24.634631    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg" event={"ID":"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606","Type":"ContainerStarted","Data":"ad38b5e0c8600442a56b5f6c6fe47eb1f14763129db0a56512034d1374cd9e83"}
Oct 30 22:27:25.645425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.645324    5031 generic.go:356] "Generic (PLEG): container finished" podID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerID="ff5c4f8d7cddf333b84b76d79bc77d9fdfcc44f69356970f8943dfcfff997581" exitCode=0
Oct 30 22:27:25.645425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.645406    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" event={"ID":"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014","Type":"ContainerDied","Data":"ff5c4f8d7cddf333b84b76d79bc77d9fdfcc44f69356970f8943dfcfff997581"}
Oct 30 22:27:25.646033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.645865    5031 scope.go:122] "RemoveContainer" containerID="ff5c4f8d7cddf333b84b76d79bc77d9fdfcc44f69356970f8943dfcfff997581"
Oct 30 22:27:25.647600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.647566    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:25.647738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.647637    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:25.647738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.647672    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:25.866273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.866231    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:25.970285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.970187    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-kube-api-access-vd5xp\" (UniqueName: \"kubernetes.io/projected/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-kube-api-access-vd5xp\") pod \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") "
Oct 30 22:27:25.970485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.970347    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-xks-ca\") pod \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") "
Oct 30 22:27:25.970557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.970506    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-openshift-ca\") pod \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\" (UID: \"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606\") "
Oct 30 22:27:25.970933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.970888    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-openshift-ca" pod "3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" (UID: "3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:25.972741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.972719    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-xks-ca" pod "3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" (UID: "3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:25.973096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:25.973067    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-kube-api-access-vd5xp" pod "3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" (UID: "3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606"). InnerVolumeSpecName "kube-api-access-vd5xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:26.072532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.072494    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:26.072532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.072527    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd5xp\" (UniqueName: \"kubernetes.io/projected/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-kube-api-access-vd5xp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:26.072532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.072537    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:26.658498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.658466    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"
Oct 30 22:27:26.658498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.658483    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg" event={"ID":"3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606","Type":"ContainerDied","Data":"ad38b5e0c8600442a56b5f6c6fe47eb1f14763129db0a56512034d1374cd9e83"}
Oct 30 22:27:26.658959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.658530    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad38b5e0c8600442a56b5f6c6fe47eb1f14763129db0a56512034d1374cd9e83"
Oct 30 22:27:26.661119 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.661083    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" event={"ID":"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014","Type":"ContainerStarted","Data":"ce9ac1a45096c29f78ff3efe7c89c9f3dc8d4ad8196c22b29551841ef1410d25"}
Oct 30 22:27:26.751080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.751045    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:27:26.752386 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.752356    5031 patch_prober.go:29] interesting pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 22:27:26.752556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:26.752415    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 22:27:27.252612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:27.252568    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"]
Oct 30 22:27:27.495278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:27.495246    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:27.751830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:27.751785    5031 patch_prober.go:29] interesting pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 22:27:27.752405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:27.751856    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 22:27:28.245638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.245594    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"]
Oct 30 22:27:28.246694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.246671    5031 cpu_manager.go:414] "containerMap: removing container" podUID="3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" containerName="revoke-keys"
Oct 30 22:27:28.246775 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.246698    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" containerName="revoke-keys"
Oct 30 22:27:28.247098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.246911    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" containerName="revoke-keys"
Oct 30 22:27:28.256242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.254863    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.334235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.311850    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf77v\" (UniqueName: \"kubernetes.io/projected/5acb7877-6598-4b94-9298-cf965f3c2b06-kube-api-access-bf77v\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.334235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.311987    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.334235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.312131    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.344067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.344036    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"]
Oct 30 22:27:28.353682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.353563    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:28.353913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.353718    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:27:28.353913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.353748    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:28.353913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.353844    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:28.356298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.355588    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 22:27:28.370868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.370839    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"]
Oct 30 22:27:28.371521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.371502    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d1820885-4e6b-40aa-9c73-f64219b771dd" containerName="payload-processing"
Oct 30 22:27:28.371521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.371520    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1820885-4e6b-40aa-9c73-f64219b771dd" containerName="payload-processing"
Oct 30 22:27:28.371696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.371670    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1820885-4e6b-40aa-9c73-f64219b771dd" containerName="payload-processing"
Oct 30 22:27:28.375155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.375139    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.466232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.455201    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1820885-4e6b-40aa-9c73-f64219b771dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1820885-4e6b-40aa-9c73-f64219b771dd-plugins-config-volume\") pod \"d1820885-4e6b-40aa-9c73-f64219b771dd\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") "
Oct 30 22:27:28.466232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.455444    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1820885-4e6b-40aa-9c73-f64219b771dd-kube-api-access-4xmb4\" (UniqueName: \"kubernetes.io/projected/d1820885-4e6b-40aa-9c73-f64219b771dd-kube-api-access-4xmb4\") pod \"d1820885-4e6b-40aa-9c73-f64219b771dd\" (UID: \"d1820885-4e6b-40aa-9c73-f64219b771dd\") "
Oct 30 22:27:28.466232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.456031    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.466232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.456178    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf77v\" (UniqueName: \"kubernetes.io/projected/5acb7877-6598-4b94-9298-cf965f3c2b06-kube-api-access-bf77v\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.466232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.456305    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.466232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.457392    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.466232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.457863    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1820885-4e6b-40aa-9c73-f64219b771dd-plugins-config-volume" pod "d1820885-4e6b-40aa-9c73-f64219b771dd" (UID: "d1820885-4e6b-40aa-9c73-f64219b771dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:28.476837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.476744    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"]
Oct 30 22:27:28.529008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.528910    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.557626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.557584    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.558159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.558130    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29gd\" (UniqueName: \"kubernetes.io/projected/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-kube-api-access-q29gd\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.558692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.558625    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.558929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.558910    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1820885-4e6b-40aa-9c73-f64219b771dd-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.559437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.559414    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1820885-4e6b-40aa-9c73-f64219b771dd-kube-api-access-4xmb4" pod "d1820885-4e6b-40aa-9c73-f64219b771dd" (UID: "d1820885-4e6b-40aa-9c73-f64219b771dd"). InnerVolumeSpecName "kube-api-access-4xmb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:28.562136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.562112    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf77v\" (UniqueName: \"kubernetes.io/projected/5acb7877-6598-4b94-9298-cf965f3c2b06-kube-api-access-bf77v\") pod \"maas-api-revoke-sub-4de2c5f7-2mfwj\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.571173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.571152    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 22:27:28.575935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.575917    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:28.662341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.659636    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129685bc-38ce-4438-974d-bb2ea0c1a743-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129685bc-38ce-4438-974d-bb2ea0c1a743-plugins-config-volume\") pod \"129685bc-38ce-4438-974d-bb2ea0c1a743\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") "
Oct 30 22:27:28.662341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.659811    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129685bc-38ce-4438-974d-bb2ea0c1a743-kube-api-access-cftlf\" (UniqueName: \"kubernetes.io/projected/129685bc-38ce-4438-974d-bb2ea0c1a743-kube-api-access-cftlf\") pod \"129685bc-38ce-4438-974d-bb2ea0c1a743\" (UID: \"129685bc-38ce-4438-974d-bb2ea0c1a743\") "
Oct 30 22:27:28.662341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.660090    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29gd\" (UniqueName: \"kubernetes.io/projected/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-kube-api-access-q29gd\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.662341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.660201    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.662341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.660364    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.662341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.660502    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmb4\" (UniqueName: \"kubernetes.io/projected/d1820885-4e6b-40aa-9c73-f64219b771dd-kube-api-access-4xmb4\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.662341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.661255    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.663327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.662971    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129685bc-38ce-4438-974d-bb2ea0c1a743-plugins-config-volume" pod "129685bc-38ce-4438-974d-bb2ea0c1a743" (UID: "129685bc-38ce-4438-974d-bb2ea0c1a743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:28.668164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.668132    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129685bc-38ce-4438-974d-bb2ea0c1a743-kube-api-access-cftlf" pod "129685bc-38ce-4438-974d-bb2ea0c1a743" (UID: "129685bc-38ce-4438-974d-bb2ea0c1a743"). InnerVolumeSpecName "kube-api-access-cftlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:28.669789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.669765    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.683807 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.683760    5031 generic.go:356] "Generic (PLEG): container finished" podID="d1820885-4e6b-40aa-9c73-f64219b771dd" containerID="a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15" exitCode=0
Oct 30 22:27:28.684136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.684112    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" event={"ID":"d1820885-4e6b-40aa-9c73-f64219b771dd","Type":"ContainerDied","Data":"a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15"}
Oct 30 22:27:28.684276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.684176    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh" event={"ID":"d1820885-4e6b-40aa-9c73-f64219b771dd","Type":"ContainerDied","Data":"aed17c76a93975d33c66e20d06ed26b38e104b6ad76f1d320a65ced752ea188a"}
Oct 30 22:27:28.684276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.684252    5031 scope.go:122] "RemoveContainer" containerID="a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15"
Oct 30 22:27:28.684443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.684427    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"
Oct 30 22:27:28.688164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.688133    5031 generic.go:356] "Generic (PLEG): container finished" podID="129685bc-38ce-4438-974d-bb2ea0c1a743" containerID="aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286" exitCode=0
Oct 30 22:27:28.688272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.688194    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" event={"ID":"129685bc-38ce-4438-974d-bb2ea0c1a743","Type":"ContainerDied","Data":"aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286"}
Oct 30 22:27:28.688316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.688282    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b" event={"ID":"129685bc-38ce-4438-974d-bb2ea0c1a743","Type":"ContainerDied","Data":"881f1b81be8e8a5e0de9eafdae06f8d75d143c147a65f6ba0e889b46264ad30f"}
Oct 30 22:27:28.688388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.688379    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"
Oct 30 22:27:28.702172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.701916    5031 scope.go:122] "RemoveContainer" containerID="a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15"
Oct 30 22:27:28.702907 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:27:28.702557    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15\": container with ID starting with a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15 not found: ID does not exist" containerID="a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15"
Oct 30 22:27:28.702907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.702592    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15"} err="failed to get container status \"a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15\": rpc error: code = NotFound desc = could not find container \"a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15\": container with ID starting with a2395145d6d3111ed9f50284287acd2e9bb83061f32bb2464c45eec126c71d15 not found: ID does not exist"
Oct 30 22:27:28.702907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.702619    5031 scope.go:122] "RemoveContainer" containerID="aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286"
Oct 30 22:27:28.740802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.740752    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29gd\" (UniqueName: \"kubernetes.io/projected/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-kube-api-access-q29gd\") pod \"maas-api-revoke-sub-064e5290-22ltd\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:28.741802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.741785    5031 scope.go:122] "RemoveContainer" containerID="aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286"
Oct 30 22:27:28.742194 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:27:28.742173    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286\": container with ID starting with aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286 not found: ID does not exist" containerID="aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286"
Oct 30 22:27:28.742344 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.742218    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286"} err="failed to get container status \"aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286\": rpc error: code = NotFound desc = could not find container \"aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286\": container with ID starting with aae24d6a08a1bd72a3ba244638e97564c33bfc7502432ac64da15807f83d8286 not found: ID does not exist"
Oct 30 22:27:28.751679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.751643    5031 patch_prober.go:29] interesting pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 22:27:28.751778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.751703    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 22:27:28.761698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.761663    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129685bc-38ce-4438-974d-bb2ea0c1a743-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.761698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.761694    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cftlf\" (UniqueName: \"kubernetes.io/projected/129685bc-38ce-4438-974d-bb2ea0c1a743-kube-api-access-cftlf\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.980075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.979995    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"]
Oct 30 22:27:28.984741 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:27:28.984713    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acb7877_6598_4b94_9298_cf965f3c2b06.slice/crio-c6cc4b4a4624b5f6d93184dbafcc02cc5f9997a9e9af391044c322c5e002ef6c WatchSource:0}: Error finding container c6cc4b4a4624b5f6d93184dbafcc02cc5f9997a9e9af391044c322c5e002ef6c: Status 404 returned error can't find the container with id c6cc4b4a4624b5f6d93184dbafcc02cc5f9997a9e9af391044c322c5e002ef6c
Oct 30 22:27:28.985055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.985041    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"]
Oct 30 22:27:28.989727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:28.989699    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh"]
Oct 30 22:27:29.033484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.033443    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:29.142851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.142784    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"]
Oct 30 22:27:29.147837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.147804    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b"]
Oct 30 22:27:29.365328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.365292    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:29.441113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.440834    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"]
Oct 30 22:27:29.448380 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:27:29.448352    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d1aec2_b781_4d7f_9ad5_bdf74ebd4e12.slice/crio-c2ec4b02cee25b2352339f29725737e966b20d712b0757fc8cae5eb50dacdf98 WatchSource:0}: Error finding container c2ec4b02cee25b2352339f29725737e966b20d712b0757fc8cae5eb50dacdf98: Status 404 returned error can't find the container with id c2ec4b02cee25b2352339f29725737e966b20d712b0757fc8cae5eb50dacdf98
Oct 30 22:27:29.698558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.698522    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj" event={"ID":"5acb7877-6598-4b94-9298-cf965f3c2b06","Type":"ContainerStarted","Data":"e7d3ebc3b564965f527d37b6256aa29b9c3e1d6849e5760581778b837fd612ce"}
Oct 30 22:27:29.698749 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.698581    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj" event={"ID":"5acb7877-6598-4b94-9298-cf965f3c2b06","Type":"ContainerStarted","Data":"c6cc4b4a4624b5f6d93184dbafcc02cc5f9997a9e9af391044c322c5e002ef6c"}
Oct 30 22:27:29.702130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.702105    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd" event={"ID":"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12","Type":"ContainerStarted","Data":"c2ec4b02cee25b2352339f29725737e966b20d712b0757fc8cae5eb50dacdf98"}
Oct 30 22:27:29.752198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.752158    5031 patch_prober.go:29] interesting pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 22:27:29.752380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:29.752247    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 22:27:30.211800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.211700    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:30.434010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.433879    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:30.646306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.646262    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129685bc-38ce-4438-974d-bb2ea0c1a743" path="/var/lib/kubelet/pods/129685bc-38ce-4438-974d-bb2ea0c1a743/volumes"
Oct 30 22:27:30.646757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.646741    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1820885-4e6b-40aa-9c73-f64219b771dd" path="/var/lib/kubelet/pods/d1820885-4e6b-40aa-9c73-f64219b771dd/volumes"
Oct 30 22:27:30.738317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.738179    5031 generic.go:356] "Generic (PLEG): container finished" podID="47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" containerID="fb00fbfabdff377197c2109730b917c04887f2212c62b8d6fc38ac4ee6090662" exitCode=0
Oct 30 22:27:30.738317 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.738266    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd" event={"ID":"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12","Type":"ContainerDied","Data":"fb00fbfabdff377197c2109730b917c04887f2212c62b8d6fc38ac4ee6090662"}
Oct 30 22:27:30.740239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.740198    5031 generic.go:356] "Generic (PLEG): container finished" podID="5acb7877-6598-4b94-9298-cf965f3c2b06" containerID="e7d3ebc3b564965f527d37b6256aa29b9c3e1d6849e5760581778b837fd612ce" exitCode=0
Oct 30 22:27:30.740373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.740312    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj" event={"ID":"5acb7877-6598-4b94-9298-cf965f3c2b06","Type":"ContainerDied","Data":"e7d3ebc3b564965f527d37b6256aa29b9c3e1d6849e5760581778b837fd612ce"}
Oct 30 22:27:30.751614 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.751586    5031 patch_prober.go:29] interesting pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 22:27:30.751734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.751638    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 22:27:30.860669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:30.860613    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj" podStartSLOduration=2.860596187 podStartE2EDuration="2.860596187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:29.833513089 +0000 UTC m=+2261.357655854" watchObservedRunningTime="2026-10-30 22:27:30.860596187 +0000 UTC m=+2262.384738994"
Oct 30 22:27:31.751729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:31.751694    5031 patch_prober.go:29] interesting pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 22:27:31.752809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:31.752778    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 22:27:31.982476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:31.982447    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:32.005256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.005150    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-openshift-ca\") pod \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") "
Oct 30 22:27:32.005256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.005249    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-kube-api-access-q29gd\" (UniqueName: \"kubernetes.io/projected/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-kube-api-access-q29gd\") pod \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") "
Oct 30 22:27:32.005480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.005450    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-xks-ca\") pod \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\" (UID: \"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12\") "
Oct 30 22:27:32.005553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.005512    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-openshift-ca" pod "47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" (UID: "47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:32.006047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.006010    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:32.007733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.007709    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-kube-api-access-q29gd" pod "47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" (UID: "47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12"). InnerVolumeSpecName "kube-api-access-q29gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:32.007865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.007801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-xks-ca" pod "47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" (UID: "47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:32.009526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.009508    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:32.107109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.107074    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5acb7877-6598-4b94-9298-cf965f3c2b06-kube-api-access-bf77v\" (UniqueName: \"kubernetes.io/projected/5acb7877-6598-4b94-9298-cf965f3c2b06-kube-api-access-bf77v\") pod \"5acb7877-6598-4b94-9298-cf965f3c2b06\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") "
Oct 30 22:27:32.107370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.107130    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-xks-ca\") pod \"5acb7877-6598-4b94-9298-cf965f3c2b06\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") "
Oct 30 22:27:32.107370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.107223    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-openshift-ca\") pod \"5acb7877-6598-4b94-9298-cf965f3c2b06\" (UID: \"5acb7877-6598-4b94-9298-cf965f3c2b06\") "
Oct 30 22:27:32.107738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.107714    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:32.107860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.107745    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29gd\" (UniqueName: \"kubernetes.io/projected/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12-kube-api-access-q29gd\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:32.107951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.107922    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-openshift-ca" pod "5acb7877-6598-4b94-9298-cf965f3c2b06" (UID: "5acb7877-6598-4b94-9298-cf965f3c2b06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:32.109928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.109901    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-xks-ca" pod "5acb7877-6598-4b94-9298-cf965f3c2b06" (UID: "5acb7877-6598-4b94-9298-cf965f3c2b06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:32.110282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.110261    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acb7877-6598-4b94-9298-cf965f3c2b06-kube-api-access-bf77v" pod "5acb7877-6598-4b94-9298-cf965f3c2b06" (UID: "5acb7877-6598-4b94-9298-cf965f3c2b06"). InnerVolumeSpecName "kube-api-access-bf77v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:32.209044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.208995    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf77v\" (UniqueName: \"kubernetes.io/projected/5acb7877-6598-4b94-9298-cf965f3c2b06-kube-api-access-bf77v\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:32.209044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.209041    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:32.209285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.209055    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb7877-6598-4b94-9298-cf965f3c2b06-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:32.266708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.266592    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:32.757600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.757570    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:27:32.758125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.758106    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:27:32.759001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.758973    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:27:32.762373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.762354    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"
Oct 30 22:27:32.762486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.762385    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd" event={"ID":"47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12","Type":"ContainerDied","Data":"c2ec4b02cee25b2352339f29725737e966b20d712b0757fc8cae5eb50dacdf98"}
Oct 30 22:27:32.762486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.762431    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ec4b02cee25b2352339f29725737e966b20d712b0757fc8cae5eb50dacdf98"
Oct 30 22:27:32.764138 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.764121    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"
Oct 30 22:27:32.764309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.764159    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj" event={"ID":"5acb7877-6598-4b94-9298-cf965f3c2b06","Type":"ContainerDied","Data":"c6cc4b4a4624b5f6d93184dbafcc02cc5f9997a9e9af391044c322c5e002ef6c"}
Oct 30 22:27:32.764309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:32.764197    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6cc4b4a4624b5f6d93184dbafcc02cc5f9997a9e9af391044c322c5e002ef6c"
Oct 30 22:27:33.460875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:33.460831    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"]
Oct 30 22:27:33.788303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:33.785303    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:34.020239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:34.020189    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:34.725160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:34.725124    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:37.048234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:37.047509    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:37.276731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:37.276697    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:37.515856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:37.515826    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:38.461310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:38.461271    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" podUID="2426ab6c-129f-4da4-b968-f508ceab1021" containerName="payload-pre-processing" containerID="cri-o://42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1" gracePeriod=25
Oct 30 22:27:39.187254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:39.186568    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:40.298607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:40.298567    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" podUID="2426ab6c-129f-4da4-b968-f508ceab1021" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:40.921343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:40.921310    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:41.211522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:41.211485    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:41.378284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:41.378233    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"]
Oct 30 22:27:41.426795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:41.426765    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-422c1029-njtgv"]
Oct 30 22:27:41.434604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:41.434577    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:41.856030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:41.855996    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:42.277805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.277715    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" containerID="cri-o://56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca" gracePeriod=45
Oct 30 22:27:42.423239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.420890    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": dial tcp 10.132.0.118:8000: connect: connection refused"
Oct 30 22:27:42.569287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.569260    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:27:42.648154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.648121    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f2b52d-1411-4d94-b209-73a3ba685c60" path="/var/lib/kubelet/pods/22f2b52d-1411-4d94-b209-73a3ba685c60/volumes"
Oct 30 22:27:42.694757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.694723    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-home\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-home\") pod \"9f1c19fa-524b-4624-b955-1e22ab1c2184\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") "
Oct 30 22:27:42.694967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.694803    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-tmp-dir\") pod \"9f1c19fa-524b-4624-b955-1e22ab1c2184\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") "
Oct 30 22:27:42.694967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.694930    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1c19fa-524b-4624-b955-1e22ab1c2184-kube-api-access-czdjw\" (UniqueName: \"kubernetes.io/projected/9f1c19fa-524b-4624-b955-1e22ab1c2184-kube-api-access-czdjw\") pod \"9f1c19fa-524b-4624-b955-1e22ab1c2184\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") "
Oct 30 22:27:42.695108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.695089    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1c19fa-524b-4624-b955-1e22ab1c2184-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1c19fa-524b-4624-b955-1e22ab1c2184-tls-certs\") pod \"9f1c19fa-524b-4624-b955-1e22ab1c2184\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") "
Oct 30 22:27:42.695171 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.695124    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-model-cache\") pod \"9f1c19fa-524b-4624-b955-1e22ab1c2184\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") "
Oct 30 22:27:42.695250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.695187    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-tmp-dir" pod "9f1c19fa-524b-4624-b955-1e22ab1c2184" (UID: "9f1c19fa-524b-4624-b955-1e22ab1c2184"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.695250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.695230    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-dshm\") pod \"9f1c19fa-524b-4624-b955-1e22ab1c2184\" (UID: \"9f1c19fa-524b-4624-b955-1e22ab1c2184\") "
Oct 30 22:27:42.695370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.695306    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-home" pod "9f1c19fa-524b-4624-b955-1e22ab1c2184" (UID: "9f1c19fa-524b-4624-b955-1e22ab1c2184"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.695429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.695396    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-model-cache" pod "9f1c19fa-524b-4624-b955-1e22ab1c2184" (UID: "9f1c19fa-524b-4624-b955-1e22ab1c2184"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.696265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.696234    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.696371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.696270    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.696371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.696287    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.697753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.697730    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1c19fa-524b-4624-b955-1e22ab1c2184-tls-certs" pod "9f1c19fa-524b-4624-b955-1e22ab1c2184" (UID: "9f1c19fa-524b-4624-b955-1e22ab1c2184"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:42.697879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.697830    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-dshm" pod "9f1c19fa-524b-4624-b955-1e22ab1c2184" (UID: "9f1c19fa-524b-4624-b955-1e22ab1c2184"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:42.697961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.697945    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1c19fa-524b-4624-b955-1e22ab1c2184-kube-api-access-czdjw" pod "9f1c19fa-524b-4624-b955-1e22ab1c2184" (UID: "9f1c19fa-524b-4624-b955-1e22ab1c2184"). InnerVolumeSpecName "kube-api-access-czdjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:42.797439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.797358    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1c19fa-524b-4624-b955-1e22ab1c2184-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.797439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.797385    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1c19fa-524b-4624-b955-1e22ab1c2184-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.797439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.797395    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdjw\" (UniqueName: \"kubernetes.io/projected/9f1c19fa-524b-4624-b955-1e22ab1c2184-kube-api-access-czdjw\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:42.877140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.877091    5031 generic.go:356] "Generic (PLEG): container finished" podID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerID="56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca" exitCode=0
Oct 30 22:27:42.877388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.877249    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" event={"ID":"9f1c19fa-524b-4624-b955-1e22ab1c2184","Type":"ContainerDied","Data":"56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca"}
Oct 30 22:27:42.877388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.877291    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm" event={"ID":"9f1c19fa-524b-4624-b955-1e22ab1c2184","Type":"ContainerDied","Data":"6e7b126a650c145b7c4aa59bf0fbbc5d58268fcdfc9f08d9db4e73d6ae6e6784"}
Oct 30 22:27:42.877388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.877322    5031 scope.go:122] "RemoveContainer" containerID="56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca"
Oct 30 22:27:42.877560 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.877525    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"
Oct 30 22:27:42.896152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.896125    5031 scope.go:122] "RemoveContainer" containerID="56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca"
Oct 30 22:27:42.896491 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:27:42.896464    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca\": container with ID starting with 56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca not found: ID does not exist" containerID="56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca"
Oct 30 22:27:42.896637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:42.896502    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca"} err="failed to get container status \"56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca\": rpc error: code = NotFound desc = could not find container \"56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca\": container with ID starting with 56438232441c6c90f4120eff01f35d599a398e05b77d5505d41433238971ccca not found: ID does not exist"
Oct 30 22:27:43.094988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:43.094954    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"]
Oct 30 22:27:43.143453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:43.143419    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-2-w1-9cff60-kserve-6674bd5f49-62wvm"]
Oct 30 22:27:43.278328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:43.278290    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:43.986465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:43.986432    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:44.677964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:44.658112    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:44.710277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:44.708597    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" path="/var/lib/kubelet/pods/9f1c19fa-524b-4624-b955-1e22ab1c2184/volumes"
Oct 30 22:27:44.951035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:44.950935    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.674700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:45.674655    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.901030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:45.900997    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:46.292998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.292963    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"]
Oct 30 22:27:46.293631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293616    5031 cpu_manager.go:414] "containerMap: removing container" podUID="47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" containerName="revoke-keys"
Oct 30 22:27:46.293702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293633    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" containerName="revoke-keys"
Oct 30 22:27:46.293702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293649    5031 cpu_manager.go:414] "containerMap: removing container" podUID="129685bc-38ce-4438-974d-bb2ea0c1a743" containerName="payload-pre-processing"
Oct 30 22:27:46.293702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293654    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129685bc-38ce-4438-974d-bb2ea0c1a743" containerName="payload-pre-processing"
Oct 30 22:27:46.293702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293680    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5acb7877-6598-4b94-9298-cf965f3c2b06" containerName="revoke-keys"
Oct 30 22:27:46.293702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293685    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acb7877-6598-4b94-9298-cf965f3c2b06" containerName="revoke-keys"
Oct 30 22:27:46.293702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293698    5031 cpu_manager.go:414] "containerMap: removing container" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main"
Oct 30 22:27:46.293702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293703    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main"
Oct 30 22:27:46.293927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293794    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5acb7877-6598-4b94-9298-cf965f3c2b06" containerName="revoke-keys"
Oct 30 22:27:46.293927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293806    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1c19fa-524b-4624-b955-1e22ab1c2184" containerName="main"
Oct 30 22:27:46.293927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293815    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" containerName="revoke-keys"
Oct 30 22:27:46.293927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.293822    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="129685bc-38ce-4438-974d-bb2ea0c1a743" containerName="payload-pre-processing"
Oct 30 22:27:46.298870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.298848    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.341111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.341073    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.341299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.341120    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9425\" (UniqueName: \"kubernetes.io/projected/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-kube-api-access-q9425\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.341358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.341299    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.360270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.360238    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"]
Oct 30 22:27:46.404165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.404130    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:46.404403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.404194    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:46.404403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.404248    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:46.404403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.404313    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:27:46.413269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.413238    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"]
Oct 30 22:27:46.421419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.421383    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.442669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.442636    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.442911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.442796    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.442911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.442827    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9425\" (UniqueName: \"kubernetes.io/projected/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-kube-api-access-q9425\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.442911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.442891    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.443094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.442931    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vxg\" (UniqueName: \"kubernetes.io/projected/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-kube-api-access-m4vxg\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.443094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.443026    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.443889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.443860    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.460970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.460938    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.466124 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.466100    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"]
Oct 30 22:27:46.507141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.507110    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9425\" (UniqueName: \"kubernetes.io/projected/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-kube-api-access-q9425\") pod \"maas-api-revoke-sub-3f0bf654-lwmsg\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.544523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.544445    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.544674 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.544624    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vxg\" (UniqueName: \"kubernetes.io/projected/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-kube-api-access-m4vxg\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.544745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.544718    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.545265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.545201    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.547321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.547295    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.609087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.609056    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vxg\" (UniqueName: \"kubernetes.io/projected/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-kube-api-access-m4vxg\") pod \"maas-api-revoke-sub-1ab0cdc6-nxdp5\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.613752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.613725    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:46.742973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.742943    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:46.926042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:46.926008    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:47.014626 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:27:47.014581    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1481dbbc_6b7e_46f8_a3ab_f2e035627c07.slice/crio-e930b4d019b88533cbb68e83931d1beb7410a4e07a1691c55187b9053d450488 WatchSource:0}: Error finding container e930b4d019b88533cbb68e83931d1beb7410a4e07a1691c55187b9053d450488: Status 404 returned error can't find the container with id e930b4d019b88533cbb68e83931d1beb7410a4e07a1691c55187b9053d450488
Oct 30 22:27:47.014827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:47.014801    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"]
Oct 30 22:27:47.141382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:47.141338    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"]
Oct 30 22:27:47.369228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:47.369174    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:47.927870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:47.927833    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5" event={"ID":"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8","Type":"ContainerStarted","Data":"d632da1e70f7e2a01bcf93800437490c70fbc12a3c9c7e8816e57c4ee7ab4572"}
Oct 30 22:27:47.928405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:47.927948    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5" event={"ID":"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8","Type":"ContainerStarted","Data":"ed9c798ebbed7d75277734baed4e300fc87cf9540c6c98d1358a1fc9a5656ae3"}
Oct 30 22:27:47.929982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:47.929953    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg" event={"ID":"1481dbbc-6b7e-46f8-a3ab-f2e035627c07","Type":"ContainerStarted","Data":"b26b64095fdd46dd9f6820e8599c3f7d4a43da3b95fd09d092d48601eca95cf2"}
Oct 30 22:27:47.930122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:47.930003    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg" event={"ID":"1481dbbc-6b7e-46f8-a3ab-f2e035627c07","Type":"ContainerStarted","Data":"e930b4d019b88533cbb68e83931d1beb7410a4e07a1691c55187b9053d450488"}
Oct 30 22:27:48.052347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:48.052280    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5" podStartSLOduration=2.052267561 podStartE2EDuration="2.052267561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:48.050103672 +0000 UTC m=+2279.574246458" watchObservedRunningTime="2026-10-30 22:27:48.052267561 +0000 UTC m=+2279.576410325"
Oct 30 22:27:48.175509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:48.175440    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg" podStartSLOduration=2.175419425 podStartE2EDuration="2.175419425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:48.168449189 +0000 UTC m=+2279.692591957" watchObservedRunningTime="2026-10-30 22:27:48.175419425 +0000 UTC m=+2279.699562191"
Oct 30 22:27:48.720725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:48.720690    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:27:48.936995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:48.936958    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:48.980304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:48.979884    5031 generic.go:356] "Generic (PLEG): container finished" podID="f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" containerID="d632da1e70f7e2a01bcf93800437490c70fbc12a3c9c7e8816e57c4ee7ab4572" exitCode=0
Oct 30 22:27:48.980304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:48.980029    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5" event={"ID":"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8","Type":"ContainerDied","Data":"d632da1e70f7e2a01bcf93800437490c70fbc12a3c9c7e8816e57c4ee7ab4572"}
Oct 30 22:27:49.002587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:49.002509    5031 generic.go:356] "Generic (PLEG): container finished" podID="1481dbbc-6b7e-46f8-a3ab-f2e035627c07" containerID="b26b64095fdd46dd9f6820e8599c3f7d4a43da3b95fd09d092d48601eca95cf2" exitCode=0
Oct 30 22:27:49.003528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:49.003494    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg" event={"ID":"1481dbbc-6b7e-46f8-a3ab-f2e035627c07","Type":"ContainerDied","Data":"b26b64095fdd46dd9f6820e8599c3f7d4a43da3b95fd09d092d48601eca95cf2"}
Oct 30 22:27:49.175226 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:49.175167    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:50.209534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.209502    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:50.218887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.218860    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-xks-ca\") pod \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") "
Oct 30 22:27:50.219048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.218921    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-openshift-ca\") pod \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") "
Oct 30 22:27:50.219129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.219059    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-kube-api-access-m4vxg\" (UniqueName: \"kubernetes.io/projected/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-kube-api-access-m4vxg\") pod \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\" (UID: \"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8\") "
Oct 30 22:27:50.219367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.219332    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-openshift-ca" pod "f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" (UID: "f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:50.219609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.219589    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.221140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.221119    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-xks-ca" pod "f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" (UID: "f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:50.221662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.221640    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-kube-api-access-m4vxg" pod "f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" (UID: "f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8"). InnerVolumeSpecName "kube-api-access-m4vxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:50.250440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.250420    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:50.298705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.298624    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" podUID="2426ab6c-129f-4da4-b968-f508ceab1021" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:50.320417 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.320388    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-kube-api-access-q9425\" (UniqueName: \"kubernetes.io/projected/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-kube-api-access-q9425\") pod \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") "
Oct 30 22:27:50.320541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.320448    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-openshift-ca\") pod \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") "
Oct 30 22:27:50.320585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.320555    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-xks-ca\") pod \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\" (UID: \"1481dbbc-6b7e-46f8-a3ab-f2e035627c07\") "
Oct 30 22:27:50.320760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.320741    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-openshift-ca" pod "1481dbbc-6b7e-46f8-a3ab-f2e035627c07" (UID: "1481dbbc-6b7e-46f8-a3ab-f2e035627c07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:50.321027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.321013    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.321083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.321029    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vxg\" (UniqueName: \"kubernetes.io/projected/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-kube-api-access-m4vxg\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.321083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.321039    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.323509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.323479    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-xks-ca" pod "1481dbbc-6b7e-46f8-a3ab-f2e035627c07" (UID: "1481dbbc-6b7e-46f8-a3ab-f2e035627c07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:50.323610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.323585    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-kube-api-access-q9425" pod "1481dbbc-6b7e-46f8-a3ab-f2e035627c07" (UID: "1481dbbc-6b7e-46f8-a3ab-f2e035627c07"). InnerVolumeSpecName "kube-api-access-q9425". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:50.422713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.422681    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9425\" (UniqueName: \"kubernetes.io/projected/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-kube-api-access-q9425\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.422713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:50.422711    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481dbbc-6b7e-46f8-a3ab-f2e035627c07-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.542777 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:27:50.542735    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:27:51.022463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.022363    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5" event={"ID":"f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8","Type":"ContainerDied","Data":"ed9c798ebbed7d75277734baed4e300fc87cf9540c6c98d1358a1fc9a5656ae3"}
Oct 30 22:27:51.022463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.022425    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9c798ebbed7d75277734baed4e300fc87cf9540c6c98d1358a1fc9a5656ae3"
Oct 30 22:27:51.022463 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.022367    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"
Oct 30 22:27:51.024285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.024254    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg" event={"ID":"1481dbbc-6b7e-46f8-a3ab-f2e035627c07","Type":"ContainerDied","Data":"e930b4d019b88533cbb68e83931d1beb7410a4e07a1691c55187b9053d450488"}
Oct 30 22:27:51.024285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.024290    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"
Oct 30 22:27:51.024513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.024306    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e930b4d019b88533cbb68e83931d1beb7410a4e07a1691c55187b9053d450488"
Oct 30 22:27:51.524112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.524077    5031 scope.go:122] "RemoveContainer" containerID="3de486e61947e90320217495beb78942ed7ca68c17dccac688edbcc1606cd920"
Oct 30 22:27:51.972679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:51.972636    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:52.251359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:52.251262    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:52.491237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:52.483166    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:53.427746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:53.427716    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.024820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.024778    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"]
Oct 30 22:27:56.025655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.025631    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" containerName="revoke-keys"
Oct 30 22:27:56.025774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.025656    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" containerName="revoke-keys"
Oct 30 22:27:56.025774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.025699    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1481dbbc-6b7e-46f8-a3ab-f2e035627c07" containerName="revoke-keys"
Oct 30 22:27:56.025774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.025708    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1481dbbc-6b7e-46f8-a3ab-f2e035627c07" containerName="revoke-keys"
Oct 30 22:27:56.025928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.025873    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" containerName="revoke-keys"
Oct 30 22:27:56.025928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.025891    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1481dbbc-6b7e-46f8-a3ab-f2e035627c07" containerName="revoke-keys"
Oct 30 22:27:56.031308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.031283    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.089645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.089615    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"]
Oct 30 22:27:56.104615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.104582    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.104854 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.104832    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.104917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.104886    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5pd\" (UniqueName: \"kubernetes.io/projected/31ccd031-5698-4634-a521-81f981e91e7e-kube-api-access-ks5pd\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.134872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.134837    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:56.135457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.135406    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:27:56.135457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.135461    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:56.136420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.136400    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:56.206594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.206551    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.206788 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.206753    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.206862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.206799    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5pd\" (UniqueName: \"kubernetes.io/projected/31ccd031-5698-4634-a521-81f981e91e7e-kube-api-access-ks5pd\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.207489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.207467    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.209308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.209285    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.269995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.269961    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5pd\" (UniqueName: \"kubernetes.io/projected/31ccd031-5698-4634-a521-81f981e91e7e-kube-api-access-ks5pd\") pod \"maas-api-revoke-sub-3a35c391-cdxkv\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.347179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.347142    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:56.750502 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:27:56.750443    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ccd031_5698_4634_a521_81f981e91e7e.slice/crio-6b38884335e88a3e011a2061786b1effb89ae80f3cbe7ddf33db31c39bbef743 WatchSource:0}: Error finding container 6b38884335e88a3e011a2061786b1effb89ae80f3cbe7ddf33db31c39bbef743: Status 404 returned error can't find the container with id 6b38884335e88a3e011a2061786b1effb89ae80f3cbe7ddf33db31c39bbef743
Oct 30 22:27:56.750668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:56.750513    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"]
Oct 30 22:27:57.080069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:57.080032    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv" event={"ID":"31ccd031-5698-4634-a521-81f981e91e7e","Type":"ContainerStarted","Data":"fa8d9e7dea2e48832c3eb6532e88f2824f1195d11e1e43a7e27bf5baef83cd59"}
Oct 30 22:27:57.080526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:57.080094    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv" event={"ID":"31ccd031-5698-4634-a521-81f981e91e7e","Type":"ContainerStarted","Data":"6b38884335e88a3e011a2061786b1effb89ae80f3cbe7ddf33db31c39bbef743"}
Oct 30 22:27:57.201007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:57.200952    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv" podStartSLOduration=2.200939382 podStartE2EDuration="2.200939382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:57.199492036 +0000 UTC m=+2288.723634819" watchObservedRunningTime="2026-10-30 22:27:57.200939382 +0000 UTC m=+2288.725082145"
Oct 30 22:27:58.089805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.089703    5031 generic.go:356] "Generic (PLEG): container finished" podID="31ccd031-5698-4634-a521-81f981e91e7e" containerID="fa8d9e7dea2e48832c3eb6532e88f2824f1195d11e1e43a7e27bf5baef83cd59" exitCode=0
Oct 30 22:27:58.089805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.089741    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv" event={"ID":"31ccd031-5698-4634-a521-81f981e91e7e","Type":"ContainerDied","Data":"fa8d9e7dea2e48832c3eb6532e88f2824f1195d11e1e43a7e27bf5baef83cd59"}
Oct 30 22:27:58.770655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.770562    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:27:58.771096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.771068    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:58.868949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.868912    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2426ab6c-129f-4da4-b968-f508ceab1021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426ab6c-129f-4da4-b968-f508ceab1021-plugins-config-volume\") pod \"2426ab6c-129f-4da4-b968-f508ceab1021\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") "
Oct 30 22:27:58.869125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.869103    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2426ab6c-129f-4da4-b968-f508ceab1021-kube-api-access-8k7p5\" (UniqueName: \"kubernetes.io/projected/2426ab6c-129f-4da4-b968-f508ceab1021-kube-api-access-8k7p5\") pod \"2426ab6c-129f-4da4-b968-f508ceab1021\" (UID: \"2426ab6c-129f-4da4-b968-f508ceab1021\") "
Oct 30 22:27:58.869349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.869323    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2426ab6c-129f-4da4-b968-f508ceab1021-plugins-config-volume" pod "2426ab6c-129f-4da4-b968-f508ceab1021" (UID: "2426ab6c-129f-4da4-b968-f508ceab1021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:58.869588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.869572    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2426ab6c-129f-4da4-b968-f508ceab1021-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.871638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.871614    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2426ab6c-129f-4da4-b968-f508ceab1021-kube-api-access-8k7p5" pod "2426ab6c-129f-4da4-b968-f508ceab1021" (UID: "2426ab6c-129f-4da4-b968-f508ceab1021"). InnerVolumeSpecName "kube-api-access-8k7p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:58.970573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:58.970534    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7p5\" (UniqueName: \"kubernetes.io/projected/2426ab6c-129f-4da4-b968-f508ceab1021-kube-api-access-8k7p5\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:59.100962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.100926    5031 generic.go:356] "Generic (PLEG): container finished" podID="2426ab6c-129f-4da4-b968-f508ceab1021" containerID="42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1" exitCode=0
Oct 30 22:27:59.101461 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.100976    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"
Oct 30 22:27:59.101461 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.101013    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" event={"ID":"2426ab6c-129f-4da4-b968-f508ceab1021","Type":"ContainerDied","Data":"42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1"}
Oct 30 22:27:59.101461 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.101075    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq" event={"ID":"2426ab6c-129f-4da4-b968-f508ceab1021","Type":"ContainerDied","Data":"459395b48a825e35c598ff145275b4c79ad6639d39476bdf94c23341e398bed2"}
Oct 30 22:27:59.101461 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.101101    5031 scope.go:122] "RemoveContainer" containerID="42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1"
Oct 30 22:27:59.115528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.115501    5031 scope.go:122] "RemoveContainer" containerID="42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1"
Oct 30 22:27:59.115891 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:27:59.115861    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1\": container with ID starting with 42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1 not found: ID does not exist" containerID="42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1"
Oct 30 22:27:59.115996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.115899    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1"} err="failed to get container status \"42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1\": rpc error: code = NotFound desc = could not find container \"42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1\": container with ID starting with 42719f10abb5924eee5e8195116cc5e15ac35ab859c52cb074935c297c3f47c1 not found: ID does not exist"
Oct 30 22:27:59.269103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.269068    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:27:59.278132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.278104    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"]
Oct 30 22:27:59.281877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.281857    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq"]
Oct 30 22:27:59.377381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.377301    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ccd031-5698-4634-a521-81f981e91e7e-kube-api-access-ks5pd\" (UniqueName: \"kubernetes.io/projected/31ccd031-5698-4634-a521-81f981e91e7e-kube-api-access-ks5pd\") pod \"31ccd031-5698-4634-a521-81f981e91e7e\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") "
Oct 30 22:27:59.377381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.377369    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-openshift-ca\") pod \"31ccd031-5698-4634-a521-81f981e91e7e\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") "
Oct 30 22:27:59.377647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.377435    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-xks-ca\") pod \"31ccd031-5698-4634-a521-81f981e91e7e\" (UID: \"31ccd031-5698-4634-a521-81f981e91e7e\") "
Oct 30 22:27:59.377758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.377731    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-openshift-ca" pod "31ccd031-5698-4634-a521-81f981e91e7e" (UID: "31ccd031-5698-4634-a521-81f981e91e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:59.378002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.377981    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:59.379686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.379665    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-xks-ca" pod "31ccd031-5698-4634-a521-81f981e91e7e" (UID: "31ccd031-5698-4634-a521-81f981e91e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:59.379814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.379786    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ccd031-5698-4634-a521-81f981e91e7e-kube-api-access-ks5pd" pod "31ccd031-5698-4634-a521-81f981e91e7e" (UID: "31ccd031-5698-4634-a521-81f981e91e7e"). InnerVolumeSpecName "kube-api-access-ks5pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:59.479270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.479236    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5pd\" (UniqueName: \"kubernetes.io/projected/31ccd031-5698-4634-a521-81f981e91e7e-kube-api-access-ks5pd\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:59.479270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:27:59.479263    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ccd031-5698-4634-a521-81f981e91e7e-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:00.114152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:00.114115    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"
Oct 30 22:28:00.114648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:00.114157    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv" event={"ID":"31ccd031-5698-4634-a521-81f981e91e7e","Type":"ContainerDied","Data":"6b38884335e88a3e011a2061786b1effb89ae80f3cbe7ddf33db31c39bbef743"}
Oct 30 22:28:00.114648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:00.114249    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b38884335e88a3e011a2061786b1effb89ae80f3cbe7ddf33db31c39bbef743"
Oct 30 22:28:00.645926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:00.645890    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2426ab6c-129f-4da4-b968-f508ceab1021" path="/var/lib/kubelet/pods/2426ab6c-129f-4da4-b968-f508ceab1021/volumes"
Oct 30 22:28:03.911639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:03.911607    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:05.232627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:05.232582    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:05.568867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:05.568780    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:06.805581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:06.805540    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:07.508241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:07.508167    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:08.429237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:08.426086    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:09.969997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:09.969960    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.213741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:10.213710    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.429532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:10.429497    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"]
Oct 30 22:28:10.448911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:10.448876    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.478876 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:10.478848    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6a0a09-9gpbw"]
Oct 30 22:28:10.647681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:10.647648    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c60101e-ad73-4939-b119-60f08ab0e91b" path="/var/lib/kubelet/pods/1c60101e-ad73-4939-b119-60f08ab0e91b/volumes"
Oct 30 22:28:10.940930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:10.940889    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:13.928108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:13.928069    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:14.737477 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:14.737440    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.768960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:15.768258    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.295635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:18.295597    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.525517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:18.525480    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.543970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:18.543938    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"]
Oct 30 22:28:18.544262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:18.544197    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy" containerID="cri-o://8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01" gracePeriod=30
Oct 30 22:28:18.786476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:18.786441    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:19.678037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:19.677999    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.159912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.159876    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"]
Oct 30 22:28:20.160859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.160838    5031 cpu_manager.go:414] "containerMap: removing container" podUID="31ccd031-5698-4634-a521-81f981e91e7e" containerName="revoke-keys"
Oct 30 22:28:20.160939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.160863    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccd031-5698-4634-a521-81f981e91e7e" containerName="revoke-keys"
Oct 30 22:28:20.160939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.160925    5031 cpu_manager.go:414] "containerMap: removing container" podUID="2426ab6c-129f-4da4-b968-f508ceab1021" containerName="payload-pre-processing"
Oct 30 22:28:20.160939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.160934    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2426ab6c-129f-4da4-b968-f508ceab1021" containerName="payload-pre-processing"
Oct 30 22:28:20.161094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.161083    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="2426ab6c-129f-4da4-b968-f508ceab1021" containerName="payload-pre-processing"
Oct 30 22:28:20.161141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.161103    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ccd031-5698-4634-a521-81f981e91e7e" containerName="revoke-keys"
Oct 30 22:28:20.165469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.165443    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.220472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.220417    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:20.233283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.233196    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:20.233647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.233624    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:20.270302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.270244    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:28:20.276091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.276057    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"]
Oct 30 22:28:20.281689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.281652    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.281851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.281701    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.281851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.281830    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8klnm\" (UniqueName: \"kubernetes.io/projected/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-kube-api-access-8klnm\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.331202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.330892    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.382669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.382629    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8klnm\" (UniqueName: \"kubernetes.io/projected/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-kube-api-access-8klnm\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.382866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.382807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.382866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.382839    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.383679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.383654    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.385575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.385547    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.466289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.465427    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8klnm\" (UniqueName: \"kubernetes.io/projected/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-kube-api-access-8klnm\") pod \"maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.486452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:20.485437    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:20.623540 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:20.623493    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:28:21.034070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:21.034033    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"]
Oct 30 22:28:21.034785 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:28:21.034755    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72ae03f_7b8f_484c_b7f4_9d8d9a5420a8.slice/crio-d30f27b5bfc9e408b2234af9ed3bba2cf1b09470caeafa7ee63420eb1869e5b4 WatchSource:0}: Error finding container d30f27b5bfc9e408b2234af9ed3bba2cf1b09470caeafa7ee63420eb1869e5b4: Status 404 returned error can't find the container with id d30f27b5bfc9e408b2234af9ed3bba2cf1b09470caeafa7ee63420eb1869e5b4
Oct 30 22:28:21.456017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:21.454326    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj" event={"ID":"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8","Type":"ContainerStarted","Data":"d59cfe05f3c4da12724fa9281a338d15434905441b2f8966fc570ee42463f28e"}
Oct 30 22:28:21.456017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:21.454546    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj" event={"ID":"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8","Type":"ContainerStarted","Data":"d30f27b5bfc9e408b2234af9ed3bba2cf1b09470caeafa7ee63420eb1869e5b4"}
Oct 30 22:28:21.579948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:21.579838    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj" podStartSLOduration=1.57982341 podStartE2EDuration="1.57982341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:21.578820671 +0000 UTC m=+2313.102963454" watchObservedRunningTime="2026-10-30 22:28:21.57982341 +0000 UTC m=+2313.103966174"
Oct 30 22:28:22.286522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:22.286397    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:22.506039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:22.505993    5031 generic.go:356] "Generic (PLEG): container finished" podID="f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" containerID="d59cfe05f3c4da12724fa9281a338d15434905441b2f8966fc570ee42463f28e" exitCode=0
Oct 30 22:28:22.506347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:22.506250    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj" event={"ID":"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8","Type":"ContainerDied","Data":"d59cfe05f3c4da12724fa9281a338d15434905441b2f8966fc570ee42463f28e"}
Oct 30 22:28:22.535235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:22.534801    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:22.776364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:22.775884    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:22.854302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:22.854263    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"]
Oct 30 22:28:22.861530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:22.861482    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9cff60-rzzvg"]
Oct 30 22:28:23.345897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:23.345827    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:23.956476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:23.956443    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:23.998781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:23.998753    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 22:28:24.097004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.096970    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-kube-api-access-k7hpn\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-kube-api-access-k7hpn\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097074    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-xks-ca\") pod \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") "
Oct 30 22:28:24.097239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097105    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-kube-api-access-8klnm\" (UniqueName: \"kubernetes.io/projected/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-kube-api-access-8klnm\") pod \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") "
Oct 30 22:28:24.097239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097127    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-data\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097178    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-socket\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097256    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-podinfo\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097291    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-credential-socket\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097322    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-envoy\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097350    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-token\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-token\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097476    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istiod-ca-cert\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.097622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097516    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-openshift-ca\") pod \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\" (UID: \"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8\") "
Oct 30 22:28:24.097743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097613    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-socket" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.097743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.097649    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-certs\") pod \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\" (UID: \"ac3999f7-0b9d-4cb8-8023-e1703c40c68b\") "
Oct 30 22:28:24.098111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098089    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-data" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.098198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098185    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-credential-socket" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.098325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098300    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-openshift-ca" pod "f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" (UID: "f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:24.098325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098308    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-certs" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.098472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098367    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istiod-ca-cert" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:24.098472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098368    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.098472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098398    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.098472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.098413    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.100448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.100419    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-podinfo" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:24.100448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.100422    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-xks-ca" pod "f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" (UID: "f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:24.100646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.100609    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-kube-api-access-k7hpn" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "kube-api-access-k7hpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:24.100956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.100939    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-token" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:24.101145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.101125    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-envoy" pod "ac3999f7-0b9d-4cb8-8023-e1703c40c68b" (UID: "ac3999f7-0b9d-4cb8-8023-e1703c40c68b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.101371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.101354    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-kube-api-access-8klnm" pod "f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" (UID: "f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8"). InnerVolumeSpecName "kube-api-access-8klnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:24.199765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199678    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7hpn\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-kube-api-access-k7hpn\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.199765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199718    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.199765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199735    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8klnm\" (UniqueName: \"kubernetes.io/projected/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-kube-api-access-8klnm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.199765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199751    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.199765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199766    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.200079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199783    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.200079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199804    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.200079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199822    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.200079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.199836    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3999f7-0b9d-4cb8-8023-e1703c40c68b-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.567106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.567009    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"
Oct 30 22:28:24.567106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.567042    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj" event={"ID":"f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8","Type":"ContainerDied","Data":"d30f27b5bfc9e408b2234af9ed3bba2cf1b09470caeafa7ee63420eb1869e5b4"}
Oct 30 22:28:24.567634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.567128    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30f27b5bfc9e408b2234af9ed3bba2cf1b09470caeafa7ee63420eb1869e5b4"
Oct 30 22:28:24.569371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.569343    5031 generic.go:356] "Generic (PLEG): container finished" podID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerID="8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01" exitCode=0
Oct 30 22:28:24.569507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.569403    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"
Oct 30 22:28:24.569507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.569425    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" event={"ID":"ac3999f7-0b9d-4cb8-8023-e1703c40c68b","Type":"ContainerDied","Data":"8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01"}
Oct 30 22:28:24.569507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.569476    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5" event={"ID":"ac3999f7-0b9d-4cb8-8023-e1703c40c68b","Type":"ContainerDied","Data":"e17c22443f7f47291b9111d4423fa27088dd389064bee09d6c98fb7e0c522d1c"}
Oct 30 22:28:24.569649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.569507    5031 scope.go:122] "RemoveContainer" containerID="8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01"
Oct 30 22:28:24.582902 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.582883    5031 scope.go:122] "RemoveContainer" containerID="5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f"
Oct 30 22:28:24.595670 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.595648    5031 scope.go:122] "RemoveContainer" containerID="8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01"
Oct 30 22:28:24.596046 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:24.596014    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01\": container with ID starting with 8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01 not found: ID does not exist" containerID="8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01"
Oct 30 22:28:24.596156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.596062    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01"} err="failed to get container status \"8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01\": rpc error: code = NotFound desc = could not find container \"8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01\": container with ID starting with 8587958924f742194140b9ee6d0bdae31ce444de1eadaee36d11484b0e28be01 not found: ID does not exist"
Oct 30 22:28:24.596156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.596089    5031 scope.go:122] "RemoveContainer" containerID="5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f"
Oct 30 22:28:24.596459 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:24.596439    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f\": container with ID starting with 5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f not found: ID does not exist" containerID="5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f"
Oct 30 22:28:24.596539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.596465    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f"} err="failed to get container status \"5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f\": rpc error: code = NotFound desc = could not find container \"5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f\": container with ID starting with 5690229f8c64a70f2f6ec9572b95f5450c82a8e3978ec1bfb2849bafaf7e5c8f not found: ID does not exist"
Oct 30 22:28:24.647623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.647591    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606" path="/var/lib/kubelet/pods/3d2f96aa-e6c6-4c4a-8167-e2e6f4f21606/volumes"
Oct 30 22:28:24.692885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.692851    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"]
Oct 30 22:28:24.759609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.759575    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"]
Oct 30 22:28:24.767390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.767359    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5"]
Oct 30 22:28:24.895997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:24.895964    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:25.167040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:25.166954    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:26.066657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:26.066621    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"]
Oct 30 22:28:26.070184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:26.070161    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058f09b6-4fljx"]
Oct 30 22:28:26.646587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:26.646553    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" path="/var/lib/kubelet/pods/ac3999f7-0b9d-4cb8-8023-e1703c40c68b/volumes"
Oct 30 22:28:26.647007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:26.646992    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e750cc-2e81-444a-b2be-fac2ca9f2e5f" path="/var/lib/kubelet/pods/f8e750cc-2e81-444a-b2be-fac2ca9f2e5f/volumes"
Oct 30 22:28:26.881457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:26.881424    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:28.102273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:28.102225    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:28.329060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:28.329022    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:28.625877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:28.625835    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:30.556154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:30.556117    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"]
Oct 30 22:28:30.803270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:30.803225    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"]
Oct 30 22:28:31.170468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:31.170432    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:31.408172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:31.408135    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:31.927496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:31.927463    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:32.165883 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:32.165828    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:32.392618 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:32.392583    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:33.535510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:33.535462    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:33.960516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:33.960479    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:34.247625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:34.247542    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:34.498482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:34.498405    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:35.557415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:35.557371    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerName="payload-processing" containerID="cri-o://386de39497ff07e183c69d5a1a72a2a97255f976cc463a7ec2c588ecf44c9a16" gracePeriod=25
Oct 30 22:28:35.803925 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:35.803875    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerName="payload-pre-processing" containerID="cri-o://bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982" gracePeriod=25
Oct 30 22:28:36.166982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:36.166941    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:36.167183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:36.166980    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:36.571791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:36.571703    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:38.606342 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:38.606297    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:40.601604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:40.601551    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main" containerID="cri-o://263b81580bd43fb53c027282a8917f7a1332572e0941ae24eb06121fbb8c7e97" gracePeriod=45
Oct 30 22:28:40.729123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:40.729089    5031 generic.go:356] "Generic (PLEG): container finished" podID="16200e83-d826-4272-95ea-a6d41611f90e" containerID="263b81580bd43fb53c027282a8917f7a1332572e0941ae24eb06121fbb8c7e97" exitCode=0
Oct 30 22:28:40.729312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:40.729163    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" event={"ID":"16200e83-d826-4272-95ea-a6d41611f90e","Type":"ContainerDied","Data":"263b81580bd43fb53c027282a8917f7a1332572e0941ae24eb06121fbb8c7e97"}
Oct 30 22:28:40.934440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:40.934403    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:28:41.050170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.050126    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-tmp-dir\") pod \"16200e83-d826-4272-95ea-a6d41611f90e\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") "
Oct 30 22:28:41.050398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.050384    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16200e83-d826-4272-95ea-a6d41611f90e-kube-api-access-vvgrt\" (UniqueName: \"kubernetes.io/projected/16200e83-d826-4272-95ea-a6d41611f90e-kube-api-access-vvgrt\") pod \"16200e83-d826-4272-95ea-a6d41611f90e\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") "
Oct 30 22:28:41.050446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.050425    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16200e83-d826-4272-95ea-a6d41611f90e-tls-certs\" (UniqueName: \"kubernetes.io/secret/16200e83-d826-4272-95ea-a6d41611f90e-tls-certs\") pod \"16200e83-d826-4272-95ea-a6d41611f90e\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") "
Oct 30 22:28:41.050496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.050466    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-tmp-dir" pod "16200e83-d826-4272-95ea-a6d41611f90e" (UID: "16200e83-d826-4272-95ea-a6d41611f90e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:41.050552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.050523    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-home\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-home\") pod \"16200e83-d826-4272-95ea-a6d41611f90e\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") "
Oct 30 22:28:41.050612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.050551    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-dshm\") pod \"16200e83-d826-4272-95ea-a6d41611f90e\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") "
Oct 30 22:28:41.050612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.050598    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-model-cache\") pod \"16200e83-d826-4272-95ea-a6d41611f90e\" (UID: \"16200e83-d826-4272-95ea-a6d41611f90e\") "
Oct 30 22:28:41.051101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.051066    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-home" pod "16200e83-d826-4272-95ea-a6d41611f90e" (UID: "16200e83-d826-4272-95ea-a6d41611f90e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:41.051101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.051080    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-model-cache" pod "16200e83-d826-4272-95ea-a6d41611f90e" (UID: "16200e83-d826-4272-95ea-a6d41611f90e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:41.051101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.051072    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.053118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.053085    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16200e83-d826-4272-95ea-a6d41611f90e-tls-certs" pod "16200e83-d826-4272-95ea-a6d41611f90e" (UID: "16200e83-d826-4272-95ea-a6d41611f90e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:41.053542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.053510    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-dshm" pod "16200e83-d826-4272-95ea-a6d41611f90e" (UID: "16200e83-d826-4272-95ea-a6d41611f90e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:41.053972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.053621    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16200e83-d826-4272-95ea-a6d41611f90e-kube-api-access-vvgrt" pod "16200e83-d826-4272-95ea-a6d41611f90e" (UID: "16200e83-d826-4272-95ea-a6d41611f90e"). InnerVolumeSpecName "kube-api-access-vvgrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:41.153263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.153153    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgrt\" (UniqueName: \"kubernetes.io/projected/16200e83-d826-4272-95ea-a6d41611f90e-kube-api-access-vvgrt\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.153445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.153269    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16200e83-d826-4272-95ea-a6d41611f90e-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.153445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.153292    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.153445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.153305    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.153445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.153318    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16200e83-d826-4272-95ea-a6d41611f90e-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.726059 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.725889    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:41.760155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.760116    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"
Oct 30 22:28:41.760365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.760151    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx" event={"ID":"16200e83-d826-4272-95ea-a6d41611f90e","Type":"ContainerDied","Data":"18354bd1d55a9f702d5aa94b8bb650499bfd22499a46b53e4a957207d41a7474"}
Oct 30 22:28:41.760365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.760249    5031 scope.go:122] "RemoveContainer" containerID="263b81580bd43fb53c027282a8917f7a1332572e0941ae24eb06121fbb8c7e97"
Oct 30 22:28:41.932514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.932475    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"]
Oct 30 22:28:41.984088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.983999    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/simulator-w2-d8837a-kserve-5bd867b954-fkcvx"]
Oct 30 22:28:41.999995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:41.998893    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:42.022260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.022222    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"]
Oct 30 22:28:42.022524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.022492    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy" containerID="cri-o://ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963" gracePeriod=30
Oct 30 22:28:42.026966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.026936    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"]
Oct 30 22:28:42.027167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.027146    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy" containerID="cri-o://57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176" gracePeriod=30
Oct 30 22:28:42.040608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.040571    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"]
Oct 30 22:28:42.040837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.040813    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy" containerID="cri-o://40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f" gracePeriod=30
Oct 30 22:28:42.227752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.227719    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:42.648002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:42.647958    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16200e83-d826-4272-95ea-a6d41611f90e" path="/var/lib/kubelet/pods/16200e83-d826-4272-95ea-a6d41611f90e/volumes"
Oct 30 22:28:43.818694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:43.818657    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:46.167060 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:46.167016    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:46.167525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:46.167035    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:47.399871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.399839    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:28:47.453693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.453662    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:28:47.457990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.457958    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:28:47.530927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.530881    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-credential-socket\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531378 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531185    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-credential-socket" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.531592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531389    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efd67607-6d80-46b0-b543-e2140d73053d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd67607-6d80-46b0-b543-e2140d73053d-istio-podinfo\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531442    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-certs\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531543    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd67607-6d80-46b0-b543-e2140d73053d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd67607-6d80-46b0-b543-e2140d73053d-istiod-ca-cert\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531585    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-data\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531632    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-istio-token\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-istio-token\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531709    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-envoy\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531741    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-socket\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.531898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.531804    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-kube-api-access-rmrxj\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-kube-api-access-rmrxj\") pod \"efd67607-6d80-46b0-b543-e2140d73053d\" (UID: \"efd67607-6d80-46b0-b543-e2140d73053d\") "
Oct 30 22:28:47.532786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.532361    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.532923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.532824    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd67607-6d80-46b0-b543-e2140d73053d-istiod-ca-cert" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.533134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.533062    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-socket" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.533472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.533433    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-certs" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.533810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.533783    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-data" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.538606 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.538584    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efd67607-6d80-46b0-b543-e2140d73053d-istio-podinfo" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:47.541910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.541876    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-istio-token" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.541910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.541884    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-envoy" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.542095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.541980    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-kube-api-access-rmrxj" pod "efd67607-6d80-46b0-b543-e2140d73053d" (UID: "efd67607-6d80-46b0-b543-e2140d73053d"). InnerVolumeSpecName "kube-api-access-rmrxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.633449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633413    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-socket\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.633449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633457    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-data\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.633751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633480    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-socket\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.633751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633511    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-credential-socket\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.633751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633550    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecde0778-eb17-47cf-8417-731378f39138-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecde0778-eb17-47cf-8417-731378f39138-istio-podinfo\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.633751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633571    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-kube-api-access-fvvvm\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-kube-api-access-fvvvm\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.633751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633615    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-kube-api-access-742kw\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-kube-api-access-742kw\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.633751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633664    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-envoy\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.633751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633699    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997a6178-1375-4f98-b1b0-4f4939da3fe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997a6178-1375-4f98-b1b0-4f4939da3fe9-istiod-ca-cert\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633757    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-podinfo\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633780    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-socket" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-data" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633784    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-socket" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633899    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-credential-socket" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633915    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-credential-socket\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.633960    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-certs\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634007    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-envoy\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634072    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-data\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634115    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-istio-token\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-istio-token\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.634149 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634147    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecde0778-eb17-47cf-8417-731378f39138-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecde0778-eb17-47cf-8417-731378f39138-istiod-ca-cert\") pod \"ecde0778-eb17-47cf-8417-731378f39138\" (UID: \"ecde0778-eb17-47cf-8417-731378f39138\") "
Oct 30 22:28:47.634802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634183    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-token\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-token\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.634802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634233    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997a6178-1375-4f98-b1b0-4f4939da3fe9-istiod-ca-cert" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.634802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634258    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-certs\") pod \"997a6178-1375-4f98-b1b0-4f4939da3fe9\" (UID: \"997a6178-1375-4f98-b1b0-4f4939da3fe9\") "
Oct 30 22:28:47.634802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634506    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-data" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.634802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634486    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-credential-socket" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.634802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634687    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-certs" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634919    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634942    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd67607-6d80-46b0-b543-e2140d73053d-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634958    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634974    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.634989    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635002    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635018    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635032    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd67607-6d80-46b0-b543-e2140d73053d-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635047    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635060    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635075    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997a6178-1375-4f98-b1b0-4f4939da3fe9-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635090    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635104    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd67607-6d80-46b0-b543-e2140d73053d-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635117    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmrxj\" (UniqueName: \"kubernetes.io/projected/efd67607-6d80-46b0-b543-e2140d73053d-kube-api-access-rmrxj\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635131    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.635154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.635143    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.637018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.636928    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-kube-api-access-fvvvm" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "kube-api-access-fvvvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.637018 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.636939    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecde0778-eb17-47cf-8417-731378f39138-istio-podinfo" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:47.637237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.637162    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-certs" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.637318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.637257    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-podinfo" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:47.637375 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.637330    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-istio-token" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.637454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.637418    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-envoy" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.637885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.637857    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecde0778-eb17-47cf-8417-731378f39138-istiod-ca-cert" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.638489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.638460    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-envoy" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:47.638911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.638890    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-token" pod "997a6178-1375-4f98-b1b0-4f4939da3fe9" (UID: "997a6178-1375-4f98-b1b0-4f4939da3fe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.639024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.639007    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-kube-api-access-742kw" pod "ecde0778-eb17-47cf-8417-731378f39138" (UID: "ecde0778-eb17-47cf-8417-731378f39138"). InnerVolumeSpecName "kube-api-access-742kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736749    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecde0778-eb17-47cf-8417-731378f39138-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736779    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvvvm\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-kube-api-access-fvvvm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736794    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742kw\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-kube-api-access-742kw\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736804    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecde0778-eb17-47cf-8417-731378f39138-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736813    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736822    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736830    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecde0778-eb17-47cf-8417-731378f39138-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736845    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecde0778-eb17-47cf-8417-731378f39138-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736859    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997a6178-1375-4f98-b1b0-4f4939da3fe9-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.736887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.736867    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997a6178-1375-4f98-b1b0-4f4939da3fe9-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.812260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.812224    5031 generic.go:356] "Generic (PLEG): container finished" podID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerID="40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f" exitCode=0
Oct 30 22:28:47.812452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.812289    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"
Oct 30 22:28:47.812452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.812309    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" event={"ID":"997a6178-1375-4f98-b1b0-4f4939da3fe9","Type":"ContainerDied","Data":"40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f"}
Oct 30 22:28:47.812452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.812377    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs" event={"ID":"997a6178-1375-4f98-b1b0-4f4939da3fe9","Type":"ContainerDied","Data":"ea4a7140535208fa5cbd693f3fb8526bffeb368173b290b6f37bd6c4bf767a9e"}
Oct 30 22:28:47.812452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.812410    5031 scope.go:122] "RemoveContainer" containerID="40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f"
Oct 30 22:28:47.814650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.814409    5031 generic.go:356] "Generic (PLEG): container finished" podID="ecde0778-eb17-47cf-8417-731378f39138" containerID="ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963" exitCode=0
Oct 30 22:28:47.814650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.814488    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" event={"ID":"ecde0778-eb17-47cf-8417-731378f39138","Type":"ContainerDied","Data":"ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963"}
Oct 30 22:28:47.814650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.814526    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh" event={"ID":"ecde0778-eb17-47cf-8417-731378f39138","Type":"ContainerDied","Data":"6e47236323221827adbcaed2de367c5e4974853b407ec5154be9179dc4cdbf7e"}
Oct 30 22:28:47.814650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.814473    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"
Oct 30 22:28:47.816412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.816392    5031 generic.go:356] "Generic (PLEG): container finished" podID="efd67607-6d80-46b0-b543-e2140d73053d" containerID="57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176" exitCode=0
Oct 30 22:28:47.816598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.816544    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" event={"ID":"efd67607-6d80-46b0-b543-e2140d73053d","Type":"ContainerDied","Data":"57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176"}
Oct 30 22:28:47.816598 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.816574    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m" event={"ID":"efd67607-6d80-46b0-b543-e2140d73053d","Type":"ContainerDied","Data":"8303598aab5c30d45c41b206ee06e15f890c6d0b828a1803ea17f5d16ff9690f"}
Oct 30 22:28:47.816776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.816636    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"
Oct 30 22:28:47.825924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.825903    5031 scope.go:122] "RemoveContainer" containerID="40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f"
Oct 30 22:28:47.826421 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:47.826272    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f\": container with ID starting with 40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f not found: ID does not exist" containerID="40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f"
Oct 30 22:28:47.826421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.826309    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f"} err="failed to get container status \"40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f\": rpc error: code = NotFound desc = could not find container \"40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f\": container with ID starting with 40dfa9489ba4d94b47e8bb229105e13f0e09e84560c0aaa8b8c9aa45e1ff261f not found: ID does not exist"
Oct 30 22:28:47.826421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.826333    5031 scope.go:122] "RemoveContainer" containerID="ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963"
Oct 30 22:28:47.837270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.837247    5031 scope.go:122] "RemoveContainer" containerID="ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963"
Oct 30 22:28:47.837557 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:47.837540    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963\": container with ID starting with ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963 not found: ID does not exist" containerID="ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963"
Oct 30 22:28:47.837611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.837566    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963"} err="failed to get container status \"ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963\": rpc error: code = NotFound desc = could not find container \"ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963\": container with ID starting with ff10ca8457001096d75c9933f9a9f349200cf70f54b60ff2e4fff4560f8cd963 not found: ID does not exist"
Oct 30 22:28:47.837611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.837584    5031 scope.go:122] "RemoveContainer" containerID="57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176"
Oct 30 22:28:47.847544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.847527    5031 scope.go:122] "RemoveContainer" containerID="57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176"
Oct 30 22:28:47.847816 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:47.847787    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176\": container with ID starting with 57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176 not found: ID does not exist" containerID="57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176"
Oct 30 22:28:47.847885 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.847823    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176"} err="failed to get container status \"57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176\": rpc error: code = NotFound desc = could not find container \"57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176\": container with ID starting with 57f0f04a98fc20bc0a588078477aeddfbd987994edd041aefd954e8f52de1176 not found: ID does not exist"
Oct 30 22:28:47.985923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.985822    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"]
Oct 30 22:28:47.991381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:47.991353    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh"]
Oct 30 22:28:48.106265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:48.106232    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"]
Oct 30 22:28:48.111827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:48.111792    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m"]
Oct 30 22:28:48.229174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:48.229141    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"]
Oct 30 22:28:48.233603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:48.233571    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs"]
Oct 30 22:28:48.657619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:48.657576    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" path="/var/lib/kubelet/pods/997a6178-1375-4f98-b1b0-4f4939da3fe9/volumes"
Oct 30 22:28:48.658080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:48.658066    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecde0778-eb17-47cf-8417-731378f39138" path="/var/lib/kubelet/pods/ecde0778-eb17-47cf-8417-731378f39138/volumes"
Oct 30 22:28:48.658510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:48.658497    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd67607-6d80-46b0-b543-e2140d73053d" path="/var/lib/kubelet/pods/efd67607-6d80-46b0-b543-e2140d73053d/volumes"
Oct 30 22:28:49.060740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:49.060649    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:50.544262 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:50.544222    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:28:52.353847 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:52.353814    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:52.870218 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:52.870187    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:28:53.482271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.482233    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"]
Oct 30 22:28:53.483182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483156    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" containerName="revoke-keys"
Oct 30 22:28:53.483287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483187    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" containerName="revoke-keys"
Oct 30 22:28:53.483287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483229    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy"
Oct 30 22:28:53.483287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483241    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy"
Oct 30 22:28:53.483287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483256    5031 cpu_manager.go:414] "containerMap: removing container" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy"
Oct 30 22:28:53.483287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483265    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy"
Oct 30 22:28:53.483287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483275    5031 cpu_manager.go:414] "containerMap: removing container" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy"
Oct 30 22:28:53.483287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483282    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy"
Oct 30 22:28:53.483523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483293    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy"
Oct 30 22:28:53.483523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483301    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy"
Oct 30 22:28:53.483523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483323    5031 cpu_manager.go:414] "containerMap: removing container" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main"
Oct 30 22:28:53.483523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483330    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main"
Oct 30 22:28:53.483523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483520    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="16200e83-d826-4272-95ea-a6d41611f90e" containerName="main"
Oct 30 22:28:53.483688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483538    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy"
Oct 30 22:28:53.483688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483556    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" containerName="revoke-keys"
Oct 30 22:28:53.483688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483567    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy"
Oct 30 22:28:53.483688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483576    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="997a6178-1375-4f98-b1b0-4f4939da3fe9" containerName="istio-proxy"
Oct 30 22:28:53.483688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483585    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd67607-6d80-46b0-b543-e2140d73053d" containerName="istio-proxy"
Oct 30 22:28:53.483688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.483595    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecde0778-eb17-47cf-8417-731378f39138" containerName="istio-proxy"
Oct 30 22:28:53.489809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.489784    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.507895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.507866    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkt84\" (UniqueName: \"kubernetes.io/projected/2a25f616-b824-4b62-ab6e-3ae17cc6f568-kube-api-access-hkt84\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.508056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.508008    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.508132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.508091    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.545393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.545354    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"]
Oct 30 22:28:53.546351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.546326    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy"
Oct 30 22:28:53.546481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.546352    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3999f7-0b9d-4cb8-8023-e1703c40c68b" containerName="istio-proxy"
Oct 30 22:28:53.551009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.550980    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"]
Oct 30 22:28:53.556299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.551106    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.598304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.588925    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:53.598304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.589049    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:28:53.598304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.591323    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:53.598304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.591477    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:53.610956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.610916    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.611150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.611100    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkt84\" (UniqueName: \"kubernetes.io/projected/2a25f616-b824-4b62-ab6e-3ae17cc6f568-kube-api-access-hkt84\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.611261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.611155    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.611321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.611282    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.611388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.611374    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlhd\" (UniqueName: \"kubernetes.io/projected/36c0d458-d1b5-44a5-928e-3f678f5fcf61-kube-api-access-hhlhd\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.611454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.611405    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.613717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.613690    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.615255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.614724    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.615424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.615406    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"]
Oct 30 22:28:53.687233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.676686    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkt84\" (UniqueName: \"kubernetes.io/projected/2a25f616-b824-4b62-ab6e-3ae17cc6f568-kube-api-access-hkt84\") pod \"maas-api-revoke-sub-75526bb9-f92pd\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.736280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.735416    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.736280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.734580    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.748035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.748000    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlhd\" (UniqueName: \"kubernetes.io/projected/36c0d458-d1b5-44a5-928e-3f678f5fcf61-kube-api-access-hhlhd\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.751479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.751156    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.764254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.759713    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.806594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.806562    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:53.821400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.821293    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlhd\" (UniqueName: \"kubernetes.io/projected/36c0d458-d1b5-44a5-928e-3f678f5fcf61-kube-api-access-hhlhd\") pod \"maas-api-revoke-sub-0a85087d-8z8nb\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:53.873770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:53.873445    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:54.237395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:54.237365    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"]
Oct 30 22:28:54.240150 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:28:54.240107    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a25f616_b824_4b62_ab6e_3ae17cc6f568.slice/crio-252b6f0223b140fd4a6448a4f4276b0636f39377f67cc0b2483e016eadcf24bb WatchSource:0}: Error finding container 252b6f0223b140fd4a6448a4f4276b0636f39377f67cc0b2483e016eadcf24bb: Status 404 returned error can't find the container with id 252b6f0223b140fd4a6448a4f4276b0636f39377f67cc0b2483e016eadcf24bb
Oct 30 22:28:54.354132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:54.354100    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"]
Oct 30 22:28:54.355343 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:28:54.355308    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c0d458_d1b5_44a5_928e_3f678f5fcf61.slice/crio-427d6c303c2d8242b9dd6c89820376a0ae14b0cc2221f0be8a92d5ff0924e38b WatchSource:0}: Error finding container 427d6c303c2d8242b9dd6c89820376a0ae14b0cc2221f0be8a92d5ff0924e38b: Status 404 returned error can't find the container with id 427d6c303c2d8242b9dd6c89820376a0ae14b0cc2221f0be8a92d5ff0924e38b
Oct 30 22:28:54.520063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:54.519983    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.899608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:54.899568    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd" event={"ID":"2a25f616-b824-4b62-ab6e-3ae17cc6f568","Type":"ContainerStarted","Data":"0e49c046ddb8bbe4d10df2c16f52b99725715db86752d7a9af7389f6146d0938"}
Oct 30 22:28:54.899874 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:54.899635    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd" event={"ID":"2a25f616-b824-4b62-ab6e-3ae17cc6f568","Type":"ContainerStarted","Data":"252b6f0223b140fd4a6448a4f4276b0636f39377f67cc0b2483e016eadcf24bb"}
Oct 30 22:28:54.901646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:54.901604    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb" event={"ID":"36c0d458-d1b5-44a5-928e-3f678f5fcf61","Type":"ContainerStarted","Data":"07c42e141315f7557197b6685b922638e885434d782f91d74d0b374588157b9a"}
Oct 30 22:28:54.901794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:54.901658    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb" event={"ID":"36c0d458-d1b5-44a5-928e-3f678f5fcf61","Type":"ContainerStarted","Data":"427d6c303c2d8242b9dd6c89820376a0ae14b0cc2221f0be8a92d5ff0924e38b"}
Oct 30 22:28:55.031238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:55.030865    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd" podStartSLOduration=2.030829973 podStartE2EDuration="2.030829973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:55.021025589 +0000 UTC m=+2346.545168385" watchObservedRunningTime="2026-10-30 22:28:55.030829973 +0000 UTC m=+2346.554972739"
Oct 30 22:28:55.149234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:55.145572    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb" podStartSLOduration=2.145552369 podStartE2EDuration="2.145552369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:55.144676285 +0000 UTC m=+2346.668819080" watchObservedRunningTime="2026-10-30 22:28:55.145552369 +0000 UTC m=+2346.669695134"
Oct 30 22:28:56.039858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.039314    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:28:56.039858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.039384    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:56.061273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.055190    5031 generic.go:356] "Generic (PLEG): container finished" podID="2a25f616-b824-4b62-ab6e-3ae17cc6f568" containerID="0e49c046ddb8bbe4d10df2c16f52b99725715db86752d7a9af7389f6146d0938" exitCode=0
Oct 30 22:28:56.061273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.055371    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd" event={"ID":"2a25f616-b824-4b62-ab6e-3ae17cc6f568","Type":"ContainerDied","Data":"0e49c046ddb8bbe4d10df2c16f52b99725715db86752d7a9af7389f6146d0938"}
Oct 30 22:28:56.091246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.082284    5031 generic.go:356] "Generic (PLEG): container finished" podID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerID="386de39497ff07e183c69d5a1a72a2a97255f976cc463a7ec2c588ecf44c9a16" exitCode=0
Oct 30 22:28:56.091246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.082413    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" event={"ID":"43c64e5b-6293-4414-b81d-9711ddda20fa","Type":"ContainerDied","Data":"386de39497ff07e183c69d5a1a72a2a97255f976cc463a7ec2c588ecf44c9a16"}
Oct 30 22:28:56.308289 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:28:56.287425    5031 logging.go:55] [core] [Channel #6731 SubChannel #6732]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.66:50052", ServerName: "10.132.0.66:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.66:50052: connect: connection refused"
Oct 30 22:28:56.309669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.309636    5031 generic.go:356] "Generic (PLEG): container finished" podID="36c0d458-d1b5-44a5-928e-3f678f5fcf61" containerID="07c42e141315f7557197b6685b922638e885434d782f91d74d0b374588157b9a" exitCode=0
Oct 30 22:28:56.309923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.309901    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb" event={"ID":"36c0d458-d1b5-44a5-928e-3f678f5fcf61","Type":"ContainerDied","Data":"07c42e141315f7557197b6685b922638e885434d782f91d74d0b374588157b9a"}
Oct 30 22:28:56.384237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.375564    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 22:28:56.482877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.481732    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c64e5b-6293-4414-b81d-9711ddda20fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c64e5b-6293-4414-b81d-9711ddda20fa-plugins-config-volume\") pod \"43c64e5b-6293-4414-b81d-9711ddda20fa\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") "
Oct 30 22:28:56.482877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.481952    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c64e5b-6293-4414-b81d-9711ddda20fa-kube-api-access-lsfq9\" (UniqueName: \"kubernetes.io/projected/43c64e5b-6293-4414-b81d-9711ddda20fa-kube-api-access-lsfq9\") pod \"43c64e5b-6293-4414-b81d-9711ddda20fa\" (UID: \"43c64e5b-6293-4414-b81d-9711ddda20fa\") "
Oct 30 22:28:56.484340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.484289    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c64e5b-6293-4414-b81d-9711ddda20fa-plugins-config-volume" pod "43c64e5b-6293-4414-b81d-9711ddda20fa" (UID: "43c64e5b-6293-4414-b81d-9711ddda20fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:56.576287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.573513    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c64e5b-6293-4414-b81d-9711ddda20fa-kube-api-access-lsfq9" pod "43c64e5b-6293-4414-b81d-9711ddda20fa" (UID: "43c64e5b-6293-4414-b81d-9711ddda20fa"). InnerVolumeSpecName "kube-api-access-lsfq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:56.595236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.591162    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c64e5b-6293-4414-b81d-9711ddda20fa-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:56.595236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.591226    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsfq9\" (UniqueName: \"kubernetes.io/projected/43c64e5b-6293-4414-b81d-9711ddda20fa-kube-api-access-lsfq9\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:56.887696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.887670    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 22:28:56.909087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.909058    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-kube-api-access-9zbzc\" (UniqueName: \"kubernetes.io/projected/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-kube-api-access-9zbzc\") pod \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") "
Oct 30 22:28:56.909283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.909265    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-plugins-config-volume\") pod \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\" (UID: \"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda\") "
Oct 30 22:28:56.909648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.909621    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-plugins-config-volume" pod "c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" (UID: "c0a56ce6-6e63-4314-a6c6-fef93d6e0fda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:56.911430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:56.911406    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-kube-api-access-9zbzc" pod "c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" (UID: "c0a56ce6-6e63-4314-a6c6-fef93d6e0fda"). InnerVolumeSpecName "kube-api-access-9zbzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:57.010720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.010680    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zbzc\" (UniqueName: \"kubernetes.io/projected/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-kube-api-access-9zbzc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.010720 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.010724    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.266901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.266783    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.66:50052\" within 1s: context deadline exceeded"
Oct 30 22:28:57.266901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.266890    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 22:28:57.268898 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.268864    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Oct 30 22:28:57.269069 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.268945    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 22:28:57.269069 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:28:57.268959    5031 logging.go:55] [core] [Channel #6733 SubChannel #6734]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.67:50052", ServerName: "10.132.0.67:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.67:50052: operation was canceled"
Oct 30 22:28:57.319291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.319252    5031 generic.go:356] "Generic (PLEG): container finished" podID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerID="bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982" exitCode=0
Oct 30 22:28:57.319492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.319306    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"
Oct 30 22:28:57.319492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.319303    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" event={"ID":"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda","Type":"ContainerDied","Data":"bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982"}
Oct 30 22:28:57.319492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.319381    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk" event={"ID":"c0a56ce6-6e63-4314-a6c6-fef93d6e0fda","Type":"ContainerDied","Data":"5fb7c975a96832a4451e34cc6dcf5ce340df599ba2f9c69e0245aca318707998"}
Oct 30 22:28:57.319492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.319421    5031 scope.go:122] "RemoveContainer" containerID="bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982"
Oct 30 22:28:57.321421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.321397    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"
Oct 30 22:28:57.321421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.321404    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc" event={"ID":"43c64e5b-6293-4414-b81d-9711ddda20fa","Type":"ContainerDied","Data":"03fb8e6d9c9ec2db26836c76296e6fcfe37a1339dec3be643c7cfaf2332f5343"}
Oct 30 22:28:57.349762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.349728    5031 scope.go:122] "RemoveContainer" containerID="bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982"
Oct 30 22:28:57.350393 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:28:57.350368    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982\": container with ID starting with bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982 not found: ID does not exist" containerID="bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982"
Oct 30 22:28:57.350526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.350405    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982"} err="failed to get container status \"bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982\": rpc error: code = NotFound desc = could not find container \"bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982\": container with ID starting with bbf61e937f7bcf3fcfe1f3461a45d6062cf33d5b5d093b27688b60f645138982 not found: ID does not exist"
Oct 30 22:28:57.350526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.350431    5031 scope.go:122] "RemoveContainer" containerID="386de39497ff07e183c69d5a1a72a2a97255f976cc463a7ec2c588ecf44c9a16"
Oct 30 22:28:57.495983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.495824    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"]
Oct 30 22:28:57.503950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.503569    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk"]
Oct 30 22:28:57.534688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.534584    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:57.537410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.537382    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:57.619424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.619390    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"]
Oct 30 22:28:57.621295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.621265    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a25f616-b824-4b62-ab6e-3ae17cc6f568-kube-api-access-hkt84\" (UniqueName: \"kubernetes.io/projected/2a25f616-b824-4b62-ab6e-3ae17cc6f568-kube-api-access-hkt84\") pod \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") "
Oct 30 22:28:57.621436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.621404    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-xks-ca\") pod \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") "
Oct 30 22:28:57.621489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.621473    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-xks-ca\") pod \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") "
Oct 30 22:28:57.621537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.621523    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c0d458-d1b5-44a5-928e-3f678f5fcf61-kube-api-access-hhlhd\" (UniqueName: \"kubernetes.io/projected/36c0d458-d1b5-44a5-928e-3f678f5fcf61-kube-api-access-hhlhd\") pod \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") "
Oct 30 22:28:57.621589 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.621560    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-openshift-ca\") pod \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\" (UID: \"36c0d458-d1b5-44a5-928e-3f678f5fcf61\") "
Oct 30 22:28:57.621644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.621619    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-openshift-ca\") pod \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\" (UID: \"2a25f616-b824-4b62-ab6e-3ae17cc6f568\") "
Oct 30 22:28:57.622296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.622140    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-openshift-ca" pod "36c0d458-d1b5-44a5-928e-3f678f5fcf61" (UID: "36c0d458-d1b5-44a5-928e-3f678f5fcf61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:57.622296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.622170    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-openshift-ca" pod "2a25f616-b824-4b62-ab6e-3ae17cc6f568" (UID: "2a25f616-b824-4b62-ab6e-3ae17cc6f568"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:57.623856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.623828    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c0d458-d1b5-44a5-928e-3f678f5fcf61-kube-api-access-hhlhd" pod "36c0d458-d1b5-44a5-928e-3f678f5fcf61" (UID: "36c0d458-d1b5-44a5-928e-3f678f5fcf61"). InnerVolumeSpecName "kube-api-access-hhlhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:57.623961 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.623939    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc"]
Oct 30 22:28:57.624076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.624057    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a25f616-b824-4b62-ab6e-3ae17cc6f568-kube-api-access-hkt84" pod "2a25f616-b824-4b62-ab6e-3ae17cc6f568" (UID: "2a25f616-b824-4b62-ab6e-3ae17cc6f568"). InnerVolumeSpecName "kube-api-access-hkt84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:57.624578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.624557    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-xks-ca" pod "2a25f616-b824-4b62-ab6e-3ae17cc6f568" (UID: "2a25f616-b824-4b62-ab6e-3ae17cc6f568"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:57.624642 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.624577    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-xks-ca" pod "36c0d458-d1b5-44a5-928e-3f678f5fcf61" (UID: "36c0d458-d1b5-44a5-928e-3f678f5fcf61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:57.723178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.723132    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkt84\" (UniqueName: \"kubernetes.io/projected/2a25f616-b824-4b62-ab6e-3ae17cc6f568-kube-api-access-hkt84\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.723178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.723174    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.723509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.723190    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.723509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.723242    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlhd\" (UniqueName: \"kubernetes.io/projected/36c0d458-d1b5-44a5-928e-3f678f5fcf61-kube-api-access-hhlhd\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.723509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.723265    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c0d458-d1b5-44a5-928e-3f678f5fcf61-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.723509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:57.723280    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a25f616-b824-4b62-ab6e-3ae17cc6f568-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:58.331259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.331155    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"
Oct 30 22:28:58.331259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.331188    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd" event={"ID":"2a25f616-b824-4b62-ab6e-3ae17cc6f568","Type":"ContainerDied","Data":"252b6f0223b140fd4a6448a4f4276b0636f39377f67cc0b2483e016eadcf24bb"}
Oct 30 22:28:58.331767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.331295    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="252b6f0223b140fd4a6448a4f4276b0636f39377f67cc0b2483e016eadcf24bb"
Oct 30 22:28:58.334183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.334155    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"
Oct 30 22:28:58.334653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.334165    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb" event={"ID":"36c0d458-d1b5-44a5-928e-3f678f5fcf61","Type":"ContainerDied","Data":"427d6c303c2d8242b9dd6c89820376a0ae14b0cc2221f0be8a92d5ff0924e38b"}
Oct 30 22:28:58.334805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.334675    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="427d6c303c2d8242b9dd6c89820376a0ae14b0cc2221f0be8a92d5ff0924e38b"
Oct 30 22:28:58.646567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.646522    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" path="/var/lib/kubelet/pods/43c64e5b-6293-4414-b81d-9711ddda20fa/volumes"
Oct 30 22:28:58.647057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:58.647035    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" path="/var/lib/kubelet/pods/c0a56ce6-6e63-4314-a6c6-fef93d6e0fda/volumes"
Oct 30 22:28:59.721919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:28:59.721885    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:01.225653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:01.225610    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:01.459032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:01.458784    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:04.999406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:04.999368    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:05.229596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:05.229559    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:06.547785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.547751    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"]
Oct 30 22:29:06.548382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548367    5031 cpu_manager.go:414] "containerMap: removing container" podUID="2a25f616-b824-4b62-ab6e-3ae17cc6f568" containerName="revoke-keys"
Oct 30 22:29:06.548452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548383    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a25f616-b824-4b62-ab6e-3ae17cc6f568" containerName="revoke-keys"
Oct 30 22:29:06.548452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548411    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerName="payload-pre-processing"
Oct 30 22:29:06.548452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548416    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerName="payload-pre-processing"
Oct 30 22:29:06.548452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548426    5031 cpu_manager.go:414] "containerMap: removing container" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerName="payload-processing"
Oct 30 22:29:06.548452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548431    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerName="payload-processing"
Oct 30 22:29:06.548452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548439    5031 cpu_manager.go:414] "containerMap: removing container" podUID="36c0d458-d1b5-44a5-928e-3f678f5fcf61" containerName="revoke-keys"
Oct 30 22:29:06.548452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548444    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c0d458-d1b5-44a5-928e-3f678f5fcf61" containerName="revoke-keys"
Oct 30 22:29:06.548691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548532    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a25f616-b824-4b62-ab6e-3ae17cc6f568" containerName="revoke-keys"
Oct 30 22:29:06.548691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548544    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a56ce6-6e63-4314-a6c6-fef93d6e0fda" containerName="payload-pre-processing"
Oct 30 22:29:06.548691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548551    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c64e5b-6293-4414-b81d-9711ddda20fa" containerName="payload-processing"
Oct 30 22:29:06.548691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.548557    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c0d458-d1b5-44a5-928e-3f678f5fcf61" containerName="revoke-keys"
Oct 30 22:29:06.553326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.553300    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.614407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.614376    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"]
Oct 30 22:29:06.656514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.656478    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t5mr\" (UniqueName: \"kubernetes.io/projected/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-kube-api-access-7t5mr\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.656716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.656532    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.656716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.656701    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.659581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.659559    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:06.659581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.659582    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:06.659850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.659682    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:06.660440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.660413    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:29:06.758080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.758040    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t5mr\" (UniqueName: \"kubernetes.io/projected/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-kube-api-access-7t5mr\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.758080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.758082    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.758443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.758314    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.758899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.758879    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.761027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.760999    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.822706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.822635    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t5mr\" (UniqueName: \"kubernetes.io/projected/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-kube-api-access-7t5mr\") pod \"maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:06.868572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:06.868528    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:07.284609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:07.284568    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"]
Oct 30 22:29:07.284919 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:29:07.284894    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc644dbb1_d3b8_46c1_b7d2_aab9ef0040ee.slice/crio-c4df3648316a7313b5ea1ce701a964ae41a4a79a26911d41f56056eaeda1cfbe WatchSource:0}: Error finding container c4df3648316a7313b5ea1ce701a964ae41a4a79a26911d41f56056eaeda1cfbe: Status 404 returned error can't find the container with id c4df3648316a7313b5ea1ce701a964ae41a4a79a26911d41f56056eaeda1cfbe
Oct 30 22:29:07.419097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:07.419063    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q" event={"ID":"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee","Type":"ContainerStarted","Data":"c4df3648316a7313b5ea1ce701a964ae41a4a79a26911d41f56056eaeda1cfbe"}
Oct 30 22:29:08.428987 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:08.428953    5031 generic.go:356] "Generic (PLEG): container finished" podID="c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" containerID="6c268284d7ee8686ad96ce46721d1c7cf6afd173cd0e9de22c01438e9919521d" exitCode=0
Oct 30 22:29:08.429423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:08.429035    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q" event={"ID":"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee","Type":"ContainerDied","Data":"6c268284d7ee8686ad96ce46721d1c7cf6afd173cd0e9de22c01438e9919521d"}
Oct 30 22:29:09.598404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.598378    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:09.693594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.693508    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-xks-ca\") pod \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") "
Oct 30 22:29:09.693761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.693687    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-kube-api-access-7t5mr\" (UniqueName: \"kubernetes.io/projected/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-kube-api-access-7t5mr\") pod \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") "
Oct 30 22:29:09.693850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.693777    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-openshift-ca\") pod \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\" (UID: \"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee\") "
Oct 30 22:29:09.694162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.694136    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-openshift-ca" pod "c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" (UID: "c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:09.694591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.694568    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:09.696032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.696008    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-kube-api-access-7t5mr" pod "c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" (UID: "c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee"). InnerVolumeSpecName "kube-api-access-7t5mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:09.696147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.696036    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-xks-ca" pod "c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" (UID: "c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:09.795770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.795737    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t5mr\" (UniqueName: \"kubernetes.io/projected/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-kube-api-access-7t5mr\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:09.795770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:09.795769    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:10.315742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:10.315697    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:10.446254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:10.446202    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"
Oct 30 22:29:10.446254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:10.446236    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q" event={"ID":"c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee","Type":"ContainerDied","Data":"c4df3648316a7313b5ea1ce701a964ae41a4a79a26911d41f56056eaeda1cfbe"}
Oct 30 22:29:10.446486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:10.446297    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4df3648316a7313b5ea1ce701a964ae41a4a79a26911d41f56056eaeda1cfbe"
Oct 30 22:29:12.417441 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:12.417410    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:15.702834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:15.702788    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:16.936981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:16.936946    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"]
Oct 30 22:29:17.050167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:17.050129    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"]
Oct 30 22:29:17.395640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:17.395603    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:17.630840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:17.630796    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:19.636999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:19.636958    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:19.858090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:19.858055    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:20.544732 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:29:20.544692    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:29:20.799293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:20.799163    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:21.937533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:21.937495    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz" podUID="32d5e446-9c52-4f08-bd66-7d04ec22899b" containerName="payload-processing" containerID="cri-o://1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee" gracePeriod=25
Oct 30 22:29:22.051309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.051258    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2" podUID="730923a4-658d-4e33-904d-9c475ed2f828" containerName="payload-pre-processing" containerID="cri-o://7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216" gracePeriod=25
Oct 30 22:29:22.223784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.223759    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:29:22.265404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.265375    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d5e446-9c52-4f08-bd66-7d04ec22899b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d5e446-9c52-4f08-bd66-7d04ec22899b-plugins-config-volume\") pod \"32d5e446-9c52-4f08-bd66-7d04ec22899b\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") "
Oct 30 22:29:22.265615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.265485    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d5e446-9c52-4f08-bd66-7d04ec22899b-kube-api-access-m6p88\" (UniqueName: \"kubernetes.io/projected/32d5e446-9c52-4f08-bd66-7d04ec22899b-kube-api-access-m6p88\") pod \"32d5e446-9c52-4f08-bd66-7d04ec22899b\" (UID: \"32d5e446-9c52-4f08-bd66-7d04ec22899b\") "
Oct 30 22:29:22.265862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.265832    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d5e446-9c52-4f08-bd66-7d04ec22899b-plugins-config-volume" pod "32d5e446-9c52-4f08-bd66-7d04ec22899b" (UID: "32d5e446-9c52-4f08-bd66-7d04ec22899b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:22.265997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.265977    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d5e446-9c52-4f08-bd66-7d04ec22899b-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:22.268412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.268377    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d5e446-9c52-4f08-bd66-7d04ec22899b-kube-api-access-m6p88" pod "32d5e446-9c52-4f08-bd66-7d04ec22899b" (UID: "32d5e446-9c52-4f08-bd66-7d04ec22899b"). InnerVolumeSpecName "kube-api-access-m6p88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:22.307186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.307163    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:29:22.366497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.366461    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730923a4-658d-4e33-904d-9c475ed2f828-kube-api-access-8fj5f\" (UniqueName: \"kubernetes.io/projected/730923a4-658d-4e33-904d-9c475ed2f828-kube-api-access-8fj5f\") pod \"730923a4-658d-4e33-904d-9c475ed2f828\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") "
Oct 30 22:29:22.366704 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.366684    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730923a4-658d-4e33-904d-9c475ed2f828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730923a4-658d-4e33-904d-9c475ed2f828-plugins-config-volume\") pod \"730923a4-658d-4e33-904d-9c475ed2f828\" (UID: \"730923a4-658d-4e33-904d-9c475ed2f828\") "
Oct 30 22:29:22.367130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.367108    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6p88\" (UniqueName: \"kubernetes.io/projected/32d5e446-9c52-4f08-bd66-7d04ec22899b-kube-api-access-m6p88\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:22.367130 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.367131    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730923a4-658d-4e33-904d-9c475ed2f828-plugins-config-volume" pod "730923a4-658d-4e33-904d-9c475ed2f828" (UID: "730923a4-658d-4e33-904d-9c475ed2f828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:22.368843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.368819    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730923a4-658d-4e33-904d-9c475ed2f828-kube-api-access-8fj5f" pod "730923a4-658d-4e33-904d-9c475ed2f828" (UID: "730923a4-658d-4e33-904d-9c475ed2f828"). InnerVolumeSpecName "kube-api-access-8fj5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:22.468076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.467993    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730923a4-658d-4e33-904d-9c475ed2f828-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:22.468076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.468032    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fj5f\" (UniqueName: \"kubernetes.io/projected/730923a4-658d-4e33-904d-9c475ed2f828-kube-api-access-8fj5f\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:22.553289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.553255    5031 generic.go:356] "Generic (PLEG): container finished" podID="32d5e446-9c52-4f08-bd66-7d04ec22899b" containerID="1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee" exitCode=0
Oct 30 22:29:22.553508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.553308    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"
Oct 30 22:29:22.553508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.553356    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz" event={"ID":"32d5e446-9c52-4f08-bd66-7d04ec22899b","Type":"ContainerDied","Data":"1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee"}
Oct 30 22:29:22.553508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.553410    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz" event={"ID":"32d5e446-9c52-4f08-bd66-7d04ec22899b","Type":"ContainerDied","Data":"7d6a0f466c075e4cf4d2d305582f6fc0349fb490d4a7bc0b8a6e759cedf5c5f0"}
Oct 30 22:29:22.553508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.553433    5031 scope.go:122] "RemoveContainer" containerID="1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee"
Oct 30 22:29:22.555616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.555357    5031 generic.go:356] "Generic (PLEG): container finished" podID="730923a4-658d-4e33-904d-9c475ed2f828" containerID="7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216" exitCode=0
Oct 30 22:29:22.555616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.555406    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2" event={"ID":"730923a4-658d-4e33-904d-9c475ed2f828","Type":"ContainerDied","Data":"7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216"}
Oct 30 22:29:22.555616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.555450    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"
Oct 30 22:29:22.555616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.555458    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2" event={"ID":"730923a4-658d-4e33-904d-9c475ed2f828","Type":"ContainerDied","Data":"8b4dc15ea0410748873599a1a34a8c10be2704215588e986c748e8b4921e8bd3"}
Oct 30 22:29:22.568452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.568431    5031 scope.go:122] "RemoveContainer" containerID="1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee"
Oct 30 22:29:22.568812 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:29:22.568791    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee\": container with ID starting with 1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee not found: ID does not exist" containerID="1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee"
Oct 30 22:29:22.568909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.568821    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee"} err="failed to get container status \"1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee\": rpc error: code = NotFound desc = could not find container \"1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee\": container with ID starting with 1d88b9c3bdee97978bee8c023eb185ef4f6fb03981d9f192b0434f3a8d841cee not found: ID does not exist"
Oct 30 22:29:22.568909 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.568842    5031 scope.go:122] "RemoveContainer" containerID="7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216"
Oct 30 22:29:22.581050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.581024    5031 scope.go:122] "RemoveContainer" containerID="7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216"
Oct 30 22:29:22.581738 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:29:22.581712    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216\": container with ID starting with 7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216 not found: ID does not exist" containerID="7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216"
Oct 30 22:29:22.581820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.581745    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216"} err="failed to get container status \"7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216\": rpc error: code = NotFound desc = could not find container \"7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216\": container with ID starting with 7befb5fd7cf08e2cbd928dd05b89667bd1110a6e03f5c8c5bcd8444531a7a216 not found: ID does not exist"
Oct 30 22:29:22.751877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.751784    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"]
Oct 30 22:29:22.759082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.759053    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz"]
Oct 30 22:29:22.871380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.871343    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"]
Oct 30 22:29:22.879103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:22.879074    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2"]
Oct 30 22:29:24.645973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:24.645937    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d5e446-9c52-4f08-bd66-7d04ec22899b" path="/var/lib/kubelet/pods/32d5e446-9c52-4f08-bd66-7d04ec22899b/volumes"
Oct 30 22:29:24.646445 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:24.646382    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730923a4-658d-4e33-904d-9c475ed2f828" path="/var/lib/kubelet/pods/730923a4-658d-4e33-904d-9c475ed2f828/volumes"
Oct 30 22:29:25.236158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:25.236115    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:27.106235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:27.106181    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:27.354633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:27.354592    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"]
Oct 30 22:29:27.405273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:27.405146    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d8837a-b2qpj"]
Oct 30 22:29:27.822884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:27.822800    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:28.648161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:28.648116    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8" path="/var/lib/kubelet/pods/f72ae03f-7b8f-484c-b7f4-9d8d9a5420a8/volumes"
Oct 30 22:29:32.546341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:32.546301    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.867984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:34.867923    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:35.125088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:35.124999    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"]
Oct 30 22:29:35.125299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:35.125275    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy" containerID="cri-o://a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271" gracePeriod=30
Oct 30 22:29:35.216311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:35.216273    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:35.468938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:35.468858    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:35.468938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:35.468920    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:35.519517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:35.519457    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:36.947300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:36.947269    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:37.449273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.449239    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"]
Oct 30 22:29:37.449858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.449830    5031 cpu_manager.go:414] "containerMap: removing container" podUID="730923a4-658d-4e33-904d-9c475ed2f828" containerName="payload-pre-processing"
Oct 30 22:29:37.449858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.449851    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730923a4-658d-4e33-904d-9c475ed2f828" containerName="payload-pre-processing"
Oct 30 22:29:37.450080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.449878    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" containerName="revoke-keys"
Oct 30 22:29:37.450080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.449883    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" containerName="revoke-keys"
Oct 30 22:29:37.450080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.449915    5031 cpu_manager.go:414] "containerMap: removing container" podUID="32d5e446-9c52-4f08-bd66-7d04ec22899b" containerName="payload-processing"
Oct 30 22:29:37.450080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.449920    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d5e446-9c52-4f08-bd66-7d04ec22899b" containerName="payload-processing"
Oct 30 22:29:37.450080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.450010    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" containerName="revoke-keys"
Oct 30 22:29:37.450080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.450027    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d5e446-9c52-4f08-bd66-7d04ec22899b" containerName="payload-processing"
Oct 30 22:29:37.450080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.450034    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="730923a4-658d-4e33-904d-9c475ed2f828" containerName="payload-pre-processing"
Oct 30 22:29:37.455085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.455063    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.511892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.511856    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"]
Oct 30 22:29:37.557846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.557818    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:37.557846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.557824    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:37.557846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.557834    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:29:37.559004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.558982    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:37.564800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.564773    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.564890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.564816    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.564890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.564864    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzs2\" (UniqueName: \"kubernetes.io/projected/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-kube-api-access-dvzs2\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.657475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.657439    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:37.666046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.666016    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.666193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.666057    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.666193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.666090    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzs2\" (UniqueName: \"kubernetes.io/projected/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-kube-api-access-dvzs2\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.666743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.666717    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.668668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.668645    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.729848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.729777    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzs2\" (UniqueName: \"kubernetes.io/projected/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-kube-api-access-dvzs2\") pod \"maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:37.772537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:37.772498    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:38.156195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:38.156161    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"]
Oct 30 22:29:38.156781 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:29:38.156759    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66a45c5_1eca_4ea8_80b9_0746bf069ffc.slice/crio-1b9c47519ccdea7ff69be0e59d7da672ea77d89f0baaf3d60f17ad807b6e3c20 WatchSource:0}: Error finding container 1b9c47519ccdea7ff69be0e59d7da672ea77d89f0baaf3d60f17ad807b6e3c20: Status 404 returned error can't find the container with id 1b9c47519ccdea7ff69be0e59d7da672ea77d89f0baaf3d60f17ad807b6e3c20
Oct 30 22:29:38.726341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:38.726192    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck" event={"ID":"e66a45c5-1eca-4ea8-80b9-0746bf069ffc","Type":"ContainerStarted","Data":"64f628f85166dc06b70e3b2f86e07dceed98c227faf1793b2c2c73a252f92ab2"}
Oct 30 22:29:38.726676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:38.726646    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck" event={"ID":"e66a45c5-1eca-4ea8-80b9-0746bf069ffc","Type":"ContainerStarted","Data":"1b9c47519ccdea7ff69be0e59d7da672ea77d89f0baaf3d60f17ad807b6e3c20"}
Oct 30 22:29:38.847827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:38.847764    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck" podStartSLOduration=1.847748369 podStartE2EDuration="1.847748369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:38.845944409 +0000 UTC m=+2390.370087192" watchObservedRunningTime="2026-10-30 22:29:38.847748369 +0000 UTC m=+2390.371891162"
Oct 30 22:29:39.735868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:39.735833    5031 generic.go:356] "Generic (PLEG): container finished" podID="e66a45c5-1eca-4ea8-80b9-0746bf069ffc" containerID="64f628f85166dc06b70e3b2f86e07dceed98c227faf1793b2c2c73a252f92ab2" exitCode=0
Oct 30 22:29:39.736330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:39.735919    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck" event={"ID":"e66a45c5-1eca-4ea8-80b9-0746bf069ffc","Type":"ContainerDied","Data":"64f628f85166dc06b70e3b2f86e07dceed98c227faf1793b2c2c73a252f92ab2"}
Oct 30 22:29:40.411751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.411663    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:29:40.503727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.503685    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istiod-ca-cert\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.503936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.503742    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-socket\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.503936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.503856    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-kube-api-access-5llk5\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-kube-api-access-5llk5\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.503936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.503916    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-podinfo\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.504110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.503982    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-credential-socket\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.504110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504017    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-certs\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.504110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504060    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-data\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.504110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504089    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-socket" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:40.504446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504114    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-envoy\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.504446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504164    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istiod-ca-cert" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:40.504446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504275    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-token\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-token\") pod \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\" (UID: \"6df4d7c2-1b7b-4f23-a112-22accd419e7d\") "
Oct 30 22:29:40.504446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504409    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-data" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:40.504679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504502    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-credential-socket" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:40.504679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504518    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-certs" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:40.504846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504819    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.504918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504857    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.504918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504871    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.504918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504885    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.504918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.504899    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.506839 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.506800    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-envoy" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:40.506839 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.506808    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-token" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:40.507049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.506888    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-podinfo" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:40.507049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.506996    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-kube-api-access-5llk5" pod "6df4d7c2-1b7b-4f23-a112-22accd419e7d" (UID: "6df4d7c2-1b7b-4f23-a112-22accd419e7d"). InnerVolumeSpecName "kube-api-access-5llk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:40.605833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.605794    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llk5\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-kube-api-access-5llk5\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.605833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.605830    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.605833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.605840    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.606101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.605852    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4d7c2-1b7b-4f23-a112-22accd419e7d-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:40.745468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.745430    5031 generic.go:356] "Generic (PLEG): container finished" podID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerID="a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271" exitCode=0
Oct 30 22:29:40.745955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.745491    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"
Oct 30 22:29:40.745955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.745521    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" event={"ID":"6df4d7c2-1b7b-4f23-a112-22accd419e7d","Type":"ContainerDied","Data":"a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271"}
Oct 30 22:29:40.745955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.745575    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw" event={"ID":"6df4d7c2-1b7b-4f23-a112-22accd419e7d","Type":"ContainerDied","Data":"4c7d2681276683ac0ad721ea4a4e66ba00999f78a014a75c33497fc8c8d3ed1c"}
Oct 30 22:29:40.745955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.745596    5031 scope.go:122] "RemoveContainer" containerID="a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271"
Oct 30 22:29:40.757794 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.757766    5031 scope.go:122] "RemoveContainer" containerID="a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271"
Oct 30 22:29:40.758257 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:29:40.758236    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271\": container with ID starting with a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271 not found: ID does not exist" containerID="a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271"
Oct 30 22:29:40.758333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.758273    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271"} err="failed to get container status \"a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271\": rpc error: code = NotFound desc = could not find container \"a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271\": container with ID starting with a31df9a6da781a8ea26926191921b5c73fb8b65d149c81b0146c3c3be9978271 not found: ID does not exist"
Oct 30 22:29:40.890782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.890753    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:40.920532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.920493    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"]
Oct 30 22:29:40.926251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:40.926201    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw"]
Oct 30 22:29:41.010098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.010004    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-openshift-ca\") pod \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") "
Oct 30 22:29:41.010303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.010105    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-kube-api-access-dvzs2\" (UniqueName: \"kubernetes.io/projected/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-kube-api-access-dvzs2\") pod \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") "
Oct 30 22:29:41.010303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.010169    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-xks-ca\") pod \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\" (UID: \"e66a45c5-1eca-4ea8-80b9-0746bf069ffc\") "
Oct 30 22:29:41.010469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.010451    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-openshift-ca" pod "e66a45c5-1eca-4ea8-80b9-0746bf069ffc" (UID: "e66a45c5-1eca-4ea8-80b9-0746bf069ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:41.010697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.010672    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:41.012588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.012562    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-xks-ca" pod "e66a45c5-1eca-4ea8-80b9-0746bf069ffc" (UID: "e66a45c5-1eca-4ea8-80b9-0746bf069ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:41.012650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.012630    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-kube-api-access-dvzs2" pod "e66a45c5-1eca-4ea8-80b9-0746bf069ffc" (UID: "e66a45c5-1eca-4ea8-80b9-0746bf069ffc"). InnerVolumeSpecName "kube-api-access-dvzs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:41.112222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.112170    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzs2\" (UniqueName: \"kubernetes.io/projected/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-kube-api-access-dvzs2\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:41.112222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.112219    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a45c5-1eca-4ea8-80b9-0746bf069ffc-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:41.756573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.756537    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"
Oct 30 22:29:41.756573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.756561    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck" event={"ID":"e66a45c5-1eca-4ea8-80b9-0746bf069ffc","Type":"ContainerDied","Data":"1b9c47519ccdea7ff69be0e59d7da672ea77d89f0baaf3d60f17ad807b6e3c20"}
Oct 30 22:29:41.757052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.756615    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9c47519ccdea7ff69be0e59d7da672ea77d89f0baaf3d60f17ad807b6e3c20"
Oct 30 22:29:41.858476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.858428    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"]
Oct 30 22:29:41.876002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:41.875962    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"]
Oct 30 22:29:42.060578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:42.060484    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:42.645862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:42.645826    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" path="/var/lib/kubelet/pods/6df4d7c2-1b7b-4f23-a112-22accd419e7d/volumes"
Oct 30 22:29:43.199793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:43.199755    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.281637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:48.281601    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.508122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:48.508082    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:49.768995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:49.768970    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:29:49.786109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:49.786077    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:29:49.876791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:49.876757    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:29:50.543268 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:29:50.543201    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:29:51.428614 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.428576    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:51.631101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.631067    5031 scope.go:122] "RemoveContainer" containerID="22ed896b54074f5ed23d85ca9279d0e4c84143bbebc71d8d82e0a5c737054fbc"
Oct 30 22:29:51.644646 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.644521    5031 scope.go:122] "RemoveContainer" containerID="9acd14060e360d49402bc4b507203a6b479c7678f80e19f8149f1e58f23259ac"
Oct 30 22:29:51.655825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.655794    5031 scope.go:122] "RemoveContainer" containerID="1b9504e2333b61aadf7d6145b0d0b06f390073d11b920aa0e0c7bcf4574144f2"
Oct 30 22:29:51.668314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.668291    5031 scope.go:122] "RemoveContainer" containerID="3a8a30f3605d22f020a0ed44f2be17ce63125d7e4fefa14cabd19f12fe0db852"
Oct 30 22:29:51.681762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.681736    5031 scope.go:122] "RemoveContainer" containerID="53bda1535f10981630f3484fe82f9323eb9837957b675ba45b6566118fe239a9"
Oct 30 22:29:51.905249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.905200    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"]
Oct 30 22:29:51.905831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.905811    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy"
Oct 30 22:29:51.905929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.905833    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy"
Oct 30 22:29:51.905929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.905885    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e66a45c5-1eca-4ea8-80b9-0746bf069ffc" containerName="revoke-keys"
Oct 30 22:29:51.905929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.905893    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66a45c5-1eca-4ea8-80b9-0746bf069ffc" containerName="revoke-keys"
Oct 30 22:29:51.906115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.906043    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66a45c5-1eca-4ea8-80b9-0746bf069ffc" containerName="revoke-keys"
Oct 30 22:29:51.906115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.906064    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df4d7c2-1b7b-4f23-a112-22accd419e7d" containerName="istio-proxy"
Oct 30 22:29:51.911271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.911248    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:51.947256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.947163    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:51.947393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.947285    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:51.947393 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.947316    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwg5\" (UniqueName: \"kubernetes.io/projected/5112b3ef-d243-47d4-8d3b-c6921f1a624b-kube-api-access-6bwg5\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:51.973233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:51.973187    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"]
Oct 30 22:29:52.018596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.018567    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:52.018596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.018585    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:52.018900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.018885    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:52.019270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.019248    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:29:52.048234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.048182    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:52.048419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.048265    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:52.048419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.048296    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwg5\" (UniqueName: \"kubernetes.io/projected/5112b3ef-d243-47d4-8d3b-c6921f1a624b-kube-api-access-6bwg5\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:52.049090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.049001    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:52.050855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.050832    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:52.112178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.112146    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwg5\" (UniqueName: \"kubernetes.io/projected/5112b3ef-d243-47d4-8d3b-c6921f1a624b-kube-api-access-6bwg5\") pod \"maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:52.225282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.225187    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:52.611406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.611363    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"]
Oct 30 22:29:52.612171 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:29:52.611950    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5112b3ef_d243_47d4_8d3b_c6921f1a624b.slice/crio-05b095cb4175451799d054c50f529e0a0b00bf5cacd5619d9930f46764cd00f8 WatchSource:0}: Error finding container 05b095cb4175451799d054c50f529e0a0b00bf5cacd5619d9930f46764cd00f8: Status 404 returned error can't find the container with id 05b095cb4175451799d054c50f529e0a0b00bf5cacd5619d9930f46764cd00f8
Oct 30 22:29:52.686385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.686329    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:52.888361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.888259    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf" event={"ID":"5112b3ef-d243-47d4-8d3b-c6921f1a624b","Type":"ContainerStarted","Data":"05b095cb4175451799d054c50f529e0a0b00bf5cacd5619d9930f46764cd00f8"}
Oct 30 22:29:52.914638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:52.914547    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.897181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:53.897144    5031 generic.go:356] "Generic (PLEG): container finished" podID="5112b3ef-d243-47d4-8d3b-c6921f1a624b" containerID="876a94dffc15de8436291b4ae0bb9edf559cc297dc0ac8696ec39af1b1e16843" exitCode=0
Oct 30 22:29:53.897849 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:53.897269    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf" event={"ID":"5112b3ef-d243-47d4-8d3b-c6921f1a624b","Type":"ContainerDied","Data":"876a94dffc15de8436291b4ae0bb9edf559cc297dc0ac8696ec39af1b1e16843"}
Oct 30 22:29:54.518315 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:54.518283    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:54.798544 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:54.798448    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:55.028352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.028315    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:55.067117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.067052    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:55.074004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.073973    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"]
Oct 30 22:29:55.085332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.085306    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-xks-ca\") pod \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") "
Oct 30 22:29:55.085499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.085351    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5112b3ef-d243-47d4-8d3b-c6921f1a624b-kube-api-access-6bwg5\" (UniqueName: \"kubernetes.io/projected/5112b3ef-d243-47d4-8d3b-c6921f1a624b-kube-api-access-6bwg5\") pod \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") "
Oct 30 22:29:55.085499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.085480    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-openshift-ca\") pod \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\" (UID: \"5112b3ef-d243-47d4-8d3b-c6921f1a624b\") "
Oct 30 22:29:55.085859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.085829    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-openshift-ca" pod "5112b3ef-d243-47d4-8d3b-c6921f1a624b" (UID: "5112b3ef-d243-47d4-8d3b-c6921f1a624b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:55.086008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.085986    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:55.088450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.088422    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5112b3ef-d243-47d4-8d3b-c6921f1a624b-kube-api-access-6bwg5" pod "5112b3ef-d243-47d4-8d3b-c6921f1a624b" (UID: "5112b3ef-d243-47d4-8d3b-c6921f1a624b"). InnerVolumeSpecName "kube-api-access-6bwg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:55.088450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.088422    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-xks-ca" pod "5112b3ef-d243-47d4-8d3b-c6921f1a624b" (UID: "5112b3ef-d243-47d4-8d3b-c6921f1a624b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:55.124977 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.124941    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d67bf9-4sh8q"]
Oct 30 22:29:55.187033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.186995    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112b3ef-d243-47d4-8d3b-c6921f1a624b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:55.187033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.187026    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bwg5\" (UniqueName: \"kubernetes.io/projected/5112b3ef-d243-47d4-8d3b-c6921f1a624b-kube-api-access-6bwg5\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:55.538083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.538046    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:55.915682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.915644    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf" event={"ID":"5112b3ef-d243-47d4-8d3b-c6921f1a624b","Type":"ContainerDied","Data":"05b095cb4175451799d054c50f529e0a0b00bf5cacd5619d9930f46764cd00f8"}
Oct 30 22:29:55.915682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.915667    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"
Oct 30 22:29:55.915929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:55.915693    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05b095cb4175451799d054c50f529e0a0b00bf5cacd5619d9930f46764cd00f8"
Oct 30 22:29:56.321480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:56.321385    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"]
Oct 30 22:29:56.642505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:56.642463    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:56.646105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:56.646074    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee" path="/var/lib/kubelet/pods/c644dbb1-d3b8-46c1-b7d2-aab9ef0040ee/volumes"
Oct 30 22:29:56.884073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:56.884002    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" containerID="cri-o://cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6" gracePeriod=45
Oct 30 22:29:56.905351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:56.905310    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" containerID="cri-o://f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a" gracePeriod=45
Oct 30 22:29:56.929450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:56.929404    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 22:29:57.174601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.174526    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:29:57.209729 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.209704    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:29:57.210807 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.210788    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/452a0cbb-a40b-4d70-972e-59ad3380495d-tls-certs\" (UniqueName: \"kubernetes.io/secret/452a0cbb-a40b-4d70-972e-59ad3380495d-tls-certs\") pod \"452a0cbb-a40b-4d70-972e-59ad3380495d\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") "
Oct 30 22:29:57.210893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.210819    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-model-cache\") pod \"452a0cbb-a40b-4d70-972e-59ad3380495d\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") "
Oct 30 22:29:57.210893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.210878    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-home\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-home\") pod \"452a0cbb-a40b-4d70-972e-59ad3380495d\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") "
Oct 30 22:29:57.211025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.210938    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-tmp-dir\") pod \"452a0cbb-a40b-4d70-972e-59ad3380495d\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") "
Oct 30 22:29:57.211025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.210995    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452a0cbb-a40b-4d70-972e-59ad3380495d-kube-api-access-kfvp2\" (UniqueName: \"kubernetes.io/projected/452a0cbb-a40b-4d70-972e-59ad3380495d-kube-api-access-kfvp2\") pod \"452a0cbb-a40b-4d70-972e-59ad3380495d\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") "
Oct 30 22:29:57.211139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.211087    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-dshm\") pod \"452a0cbb-a40b-4d70-972e-59ad3380495d\" (UID: \"452a0cbb-a40b-4d70-972e-59ad3380495d\") "
Oct 30 22:29:57.211222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.211172    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-home" pod "452a0cbb-a40b-4d70-972e-59ad3380495d" (UID: "452a0cbb-a40b-4d70-972e-59ad3380495d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.211222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.211151    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-model-cache" pod "452a0cbb-a40b-4d70-972e-59ad3380495d" (UID: "452a0cbb-a40b-4d70-972e-59ad3380495d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.211322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.211250    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-tmp-dir" pod "452a0cbb-a40b-4d70-972e-59ad3380495d" (UID: "452a0cbb-a40b-4d70-972e-59ad3380495d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.211612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.211587    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.211612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.211612    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.211777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.211626    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.213471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.213447    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-dshm" pod "452a0cbb-a40b-4d70-972e-59ad3380495d" (UID: "452a0cbb-a40b-4d70-972e-59ad3380495d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.213554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.213506    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452a0cbb-a40b-4d70-972e-59ad3380495d-tls-certs" pod "452a0cbb-a40b-4d70-972e-59ad3380495d" (UID: "452a0cbb-a40b-4d70-972e-59ad3380495d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:57.213554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.213516    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452a0cbb-a40b-4d70-972e-59ad3380495d-kube-api-access-kfvp2" pod "452a0cbb-a40b-4d70-972e-59ad3380495d" (UID: "452a0cbb-a40b-4d70-972e-59ad3380495d"). InnerVolumeSpecName "kube-api-access-kfvp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:57.312450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.312396    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86265ef-a35f-4c8b-9e28-59480c6d3657-kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/b86265ef-a35f-4c8b-9e28-59480c6d3657-kube-api-access-2qkq8\") pod \"b86265ef-a35f-4c8b-9e28-59480c6d3657\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") "
Oct 30 22:29:57.312656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.312465    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b86265ef-a35f-4c8b-9e28-59480c6d3657-tls-certs\" (UniqueName: \"kubernetes.io/secret/b86265ef-a35f-4c8b-9e28-59480c6d3657-tls-certs\") pod \"b86265ef-a35f-4c8b-9e28-59480c6d3657\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") "
Oct 30 22:29:57.312656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.312632    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-model-cache\") pod \"b86265ef-a35f-4c8b-9e28-59480c6d3657\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") "
Oct 30 22:29:57.313686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.313033    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-model-cache" pod "b86265ef-a35f-4c8b-9e28-59480c6d3657" (UID: "b86265ef-a35f-4c8b-9e28-59480c6d3657"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.313686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.313373    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-dshm\") pod \"b86265ef-a35f-4c8b-9e28-59480c6d3657\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") "
Oct 30 22:29:57.313686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.313430    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-home\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-home\") pod \"b86265ef-a35f-4c8b-9e28-59480c6d3657\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") "
Oct 30 22:29:57.313686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.313517    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-tmp-dir\") pod \"b86265ef-a35f-4c8b-9e28-59480c6d3657\" (UID: \"b86265ef-a35f-4c8b-9e28-59480c6d3657\") "
Oct 30 22:29:57.314077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.314056    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452a0cbb-a40b-4d70-972e-59ad3380495d-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.314147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.314084    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452a0cbb-a40b-4d70-972e-59ad3380495d-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.314147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.314098    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.314147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.314115    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfvp2\" (UniqueName: \"kubernetes.io/projected/452a0cbb-a40b-4d70-972e-59ad3380495d-kube-api-access-kfvp2\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.314625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.314371    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-tmp-dir" pod "b86265ef-a35f-4c8b-9e28-59480c6d3657" (UID: "b86265ef-a35f-4c8b-9e28-59480c6d3657"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.314625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.314587    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-home" pod "b86265ef-a35f-4c8b-9e28-59480c6d3657" (UID: "b86265ef-a35f-4c8b-9e28-59480c6d3657"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.317154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.317122    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-dshm" pod "b86265ef-a35f-4c8b-9e28-59480c6d3657" (UID: "b86265ef-a35f-4c8b-9e28-59480c6d3657"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:57.317517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.317490    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86265ef-a35f-4c8b-9e28-59480c6d3657-kube-api-access-2qkq8" pod "b86265ef-a35f-4c8b-9e28-59480c6d3657" (UID: "b86265ef-a35f-4c8b-9e28-59480c6d3657"). InnerVolumeSpecName "kube-api-access-2qkq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:57.317809 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.317775    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86265ef-a35f-4c8b-9e28-59480c6d3657-tls-certs" pod "b86265ef-a35f-4c8b-9e28-59480c6d3657" (UID: "b86265ef-a35f-4c8b-9e28-59480c6d3657"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:57.416917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.415907    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.416917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.415942    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.416917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.415955    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b86265ef-a35f-4c8b-9e28-59480c6d3657-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.416917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.415968    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/b86265ef-a35f-4c8b-9e28-59480c6d3657-kube-api-access-2qkq8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.416917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.415980    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b86265ef-a35f-4c8b-9e28-59480c6d3657-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.934477 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.934439    5031 generic.go:356] "Generic (PLEG): container finished" podID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerID="cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6" exitCode=0
Oct 30 22:29:57.934665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.934484    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"
Oct 30 22:29:57.934665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.934524    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" event={"ID":"452a0cbb-a40b-4d70-972e-59ad3380495d","Type":"ContainerDied","Data":"cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6"}
Oct 30 22:29:57.934665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.934574    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4" event={"ID":"452a0cbb-a40b-4d70-972e-59ad3380495d","Type":"ContainerDied","Data":"428259637b37bb987340a122fb6c78d2c954f3cd460cdbf8af0852853512c83c"}
Oct 30 22:29:57.934665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.934597    5031 scope.go:122] "RemoveContainer" containerID="cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6"
Oct 30 22:29:57.936603 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.936566    5031 generic.go:356] "Generic (PLEG): container finished" podID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerID="f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a" exitCode=0
Oct 30 22:29:57.936756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.936637    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"
Oct 30 22:29:57.936756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.936665    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" event={"ID":"b86265ef-a35f-4c8b-9e28-59480c6d3657","Type":"ContainerDied","Data":"f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a"}
Oct 30 22:29:57.936756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.936716    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" event={"ID":"b86265ef-a35f-4c8b-9e28-59480c6d3657","Type":"ContainerDied","Data":"43e8cbafd1aa17c9c2950e451c57ed40b5f0a4142bfa8c52cfbc475a13528259"}
Oct 30 22:29:57.954841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.954815    5031 scope.go:122] "RemoveContainer" containerID="cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6"
Oct 30 22:29:57.955381 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:29:57.955351    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6\": container with ID starting with cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6 not found: ID does not exist" containerID="cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6"
Oct 30 22:29:57.955530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.955392    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6"} err="failed to get container status \"cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6\": rpc error: code = NotFound desc = could not find container \"cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6\": container with ID starting with cbe21eb7dd2881de4daf51ffd969b349f1e5ddb5e85a57ecd1dfa6b6eb4c5cc6 not found: ID does not exist"
Oct 30 22:29:57.955530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.955423    5031 scope.go:122] "RemoveContainer" containerID="f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a"
Oct 30 22:29:57.968050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.968026    5031 scope.go:122] "RemoveContainer" containerID="f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a"
Oct 30 22:29:57.968401 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:29:57.968379    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a\": container with ID starting with f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a not found: ID does not exist" containerID="f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a"
Oct 30 22:29:57.968494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:57.968409    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a"} err="failed to get container status \"f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a\": rpc error: code = NotFound desc = could not find container \"f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a\": container with ID starting with f1bc6e40ecf60b7a21d926d94a3962f8db2da1071fd22ac39d8c926d992da26a not found: ID does not exist"
Oct 30 22:29:58.123986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:58.123950    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"]
Oct 30 22:29:58.136442 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:58.136408    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp"]
Oct 30 22:29:58.182856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:58.182758    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2182a5/simulator-w6-2182a5-kserve-765d949cf8-p85wp" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:58.288839 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:58.288802    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"]
Oct 30 22:29:58.296823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:58.296790    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2182a5/premium-w6-2182a5-kserve-84bc64fb59-pw4c4"]
Oct 30 22:29:58.646613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:58.646577    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" path="/var/lib/kubelet/pods/452a0cbb-a40b-4d70-972e-59ad3380495d/volumes"
Oct 30 22:29:58.647041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:29:58.646998    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" path="/var/lib/kubelet/pods/b86265ef-a35f-4c8b-9e28-59480c6d3657/volumes"
Oct 30 22:30:00.161709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.161659    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"]
Oct 30 22:30:00.162252 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162237    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main"
Oct 30 22:30:00.162311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162254    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main"
Oct 30 22:30:00.162311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162273    5031 cpu_manager.go:414] "containerMap: removing container" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main"
Oct 30 22:30:00.162311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162278    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main"
Oct 30 22:30:00.162410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162310    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5112b3ef-d243-47d4-8d3b-c6921f1a624b" containerName="revoke-keys"
Oct 30 22:30:00.162410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162316    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5112b3ef-d243-47d4-8d3b-c6921f1a624b" containerName="revoke-keys"
Oct 30 22:30:00.162410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162400    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5112b3ef-d243-47d4-8d3b-c6921f1a624b" containerName="revoke-keys"
Oct 30 22:30:00.162410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162409    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86265ef-a35f-4c8b-9e28-59480c6d3657" containerName="main"
Oct 30 22:30:00.162543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.162416    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="452a0cbb-a40b-4d70-972e-59ad3380495d" containerName="main"
Oct 30 22:30:00.167725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.167700    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.168300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.168280    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"]
Oct 30 22:30:00.172191 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.172172    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"]
Oct 30 22:30:00.172313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.172299    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.176276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.176260    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.222468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.222434    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:00.222672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.222654    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:30:00.222735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.222679    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:00.247530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.247490    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32cfce6-24a9-4ed6-bece-ead7a13b5205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.247745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.247546    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-gr96l\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.247745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.247644    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sssp\" (UniqueName: \"kubernetes.io/projected/c32cfce6-24a9-4ed6-bece-ead7a13b5205-kube-api-access-8sssp\") pod \"maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.247745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.247734    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec330f8c-d455-48b0-a4e6-ec008d31d305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.247932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.247870    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mssr\" (UniqueName: \"kubernetes.io/projected/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-kube-api-access-5mssr\") pod \"maas-api-key-cleanup-29889990-gr96l\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.247932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.247918    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7jf\" (UniqueName: \"kubernetes.io/projected/ec330f8c-d455-48b0-a4e6-ec008d31d305-kube-api-access-cs7jf\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.277796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.277754    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"]
Oct 30 22:30:00.283939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.283900    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"]
Oct 30 22:30:00.290376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.290342    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"]
Oct 30 22:30:00.348746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.348705    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec330f8c-d455-48b0-a4e6-ec008d31d305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.348978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.348775    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mssr\" (UniqueName: \"kubernetes.io/projected/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-kube-api-access-5mssr\") pod \"maas-api-key-cleanup-29889990-gr96l\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.348978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.348798    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7jf\" (UniqueName: \"kubernetes.io/projected/ec330f8c-d455-48b0-a4e6-ec008d31d305-kube-api-access-cs7jf\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.348978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.348859    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32cfce6-24a9-4ed6-bece-ead7a13b5205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.349275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.348991    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-gr96l\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.349275 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.349034    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sssp\" (UniqueName: \"kubernetes.io/projected/c32cfce6-24a9-4ed6-bece-ead7a13b5205-kube-api-access-8sssp\") pod \"maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.349582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.349558    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec330f8c-d455-48b0-a4e6-ec008d31d305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.349666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.349637    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32cfce6-24a9-4ed6-bece-ead7a13b5205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.349753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.349730    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-gr96l\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.417295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.414191    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7jf\" (UniqueName: \"kubernetes.io/projected/ec330f8c-d455-48b0-a4e6-ec008d31d305-kube-api-access-cs7jf\") pod \"maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.417295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.414407    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sssp\" (UniqueName: \"kubernetes.io/projected/c32cfce6-24a9-4ed6-bece-ead7a13b5205-kube-api-access-8sssp\") pod \"maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.417295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.416885    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:00.430100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.430067    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mssr\" (UniqueName: \"kubernetes.io/projected/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-kube-api-access-5mssr\") pod \"maas-api-key-cleanup-29889990-gr96l\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.483368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.483328    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:00.492398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.492369    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:00.498611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.498587    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:00.894036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.894003    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"]
Oct 30 22:30:00.894632 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:30:00.894570    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec330f8c_d455_48b0_a4e6_ec008d31d305.slice/crio-4134c9835ff1c92c9d6f1ff77ae80e55ddec99ed97f6823be45c0b9f58d277b9 WatchSource:0}: Error finding container 4134c9835ff1c92c9d6f1ff77ae80e55ddec99ed97f6823be45c0b9f58d277b9: Status 404 returned error can't find the container with id 4134c9835ff1c92c9d6f1ff77ae80e55ddec99ed97f6823be45c0b9f58d277b9
Oct 30 22:30:00.944957 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:30:00.944926    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d2a1f3_15f5_497a_8f2a_68c9ff261f2a.slice/crio-b84e757f85b77686475819fb7479f0b6b1585535a723ca0c45fc000dda684023 WatchSource:0}: Error finding container b84e757f85b77686475819fb7479f0b6b1585535a723ca0c45fc000dda684023: Status 404 returned error can't find the container with id b84e757f85b77686475819fb7479f0b6b1585535a723ca0c45fc000dda684023
Oct 30 22:30:00.945117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.944993    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"]
Oct 30 22:30:00.945949 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:30:00.945926    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32cfce6_24a9_4ed6_bece_ead7a13b5205.slice/crio-20b3ddb95b7a532a78e901cf98e10f7e858897fd8b7c0b94ff9cfe52adb5f593 WatchSource:0}: Error finding container 20b3ddb95b7a532a78e901cf98e10f7e858897fd8b7c0b94ff9cfe52adb5f593: Status 404 returned error can't find the container with id 20b3ddb95b7a532a78e901cf98e10f7e858897fd8b7c0b94ff9cfe52adb5f593
Oct 30 22:30:00.950145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.950117    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"]
Oct 30 22:30:00.968014 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.967984    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" event={"ID":"c32cfce6-24a9-4ed6-bece-ead7a13b5205","Type":"ContainerStarted","Data":"20b3ddb95b7a532a78e901cf98e10f7e858897fd8b7c0b94ff9cfe52adb5f593"}
Oct 30 22:30:00.970100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.970072    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerStarted","Data":"4134c9835ff1c92c9d6f1ff77ae80e55ddec99ed97f6823be45c0b9f58d277b9"}
Oct 30 22:30:00.972444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:00.972414    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerStarted","Data":"b84e757f85b77686475819fb7479f0b6b1585535a723ca0c45fc000dda684023"}
Oct 30 22:30:01.982854 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:01.982811    5031 generic.go:356] "Generic (PLEG): container finished" podID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerID="c93418383e4ad18ad9c811336f6929aa79f68b32c453e7aaa5c51e071d714a97" exitCode=60
Oct 30 22:30:01.983319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:01.982891    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" event={"ID":"c32cfce6-24a9-4ed6-bece-ead7a13b5205","Type":"ContainerDied","Data":"c93418383e4ad18ad9c811336f6929aa79f68b32c453e7aaa5c51e071d714a97"}
Oct 30 22:30:01.983319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:01.983245    5031 scope.go:122] "RemoveContainer" containerID="c93418383e4ad18ad9c811336f6929aa79f68b32c453e7aaa5c51e071d714a97"
Oct 30 22:30:01.985062 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:01.985039    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerStarted","Data":"366996d3466f3fbf6165701c1f5afd3e29600028b55b4d074b0db277856e91aa"}
Oct 30 22:30:01.988145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:01.988120    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerStarted","Data":"84c68a7a7a20e9747044ace6929b8a83e4727a83103d484d5a764548eda8f22c"}
Oct 30 22:30:02.219498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.219448    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" podStartSLOduration=2.219433971 podStartE2EDuration="2.219433971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.218820758 +0000 UTC m=+2413.742963541" watchObservedRunningTime="2026-10-30 22:30:02.219433971 +0000 UTC m=+2413.743576732"
Oct 30 22:30:02.296912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.296871    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"]
Oct 30 22:30:02.345763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.345657    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" podStartSLOduration=2.34564353 podStartE2EDuration="2.34564353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.345068824 +0000 UTC m=+2413.869211615" watchObservedRunningTime="2026-10-30 22:30:02.34564353 +0000 UTC m=+2413.869786294"
Oct 30 22:30:02.740599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.740305    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.974710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.974675    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.999155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.999062    5031 generic.go:356] "Generic (PLEG): container finished" podID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerID="366996d3466f3fbf6165701c1f5afd3e29600028b55b4d074b0db277856e91aa" exitCode=60
Oct 30 22:30:02.999615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.999153    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerDied","Data":"366996d3466f3fbf6165701c1f5afd3e29600028b55b4d074b0db277856e91aa"}
Oct 30 22:30:02.999615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:02.999505    5031 scope.go:122] "RemoveContainer" containerID="366996d3466f3fbf6165701c1f5afd3e29600028b55b4d074b0db277856e91aa"
Oct 30 22:30:03.002269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:03.002233    5031 generic.go:356] "Generic (PLEG): container finished" podID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerID="84c68a7a7a20e9747044ace6929b8a83e4727a83103d484d5a764548eda8f22c" exitCode=60
Oct 30 22:30:03.002403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:03.002274    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerDied","Data":"84c68a7a7a20e9747044ace6929b8a83e4727a83103d484d5a764548eda8f22c"}
Oct 30 22:30:03.002610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:03.002593    5031 scope.go:122] "RemoveContainer" containerID="84c68a7a7a20e9747044ace6929b8a83e4727a83103d484d5a764548eda8f22c"
Oct 30 22:30:03.004936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:03.004913    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" event={"ID":"c32cfce6-24a9-4ed6-bece-ead7a13b5205","Type":"ContainerStarted","Data":"0b083618239ab08d68f0dbd690fa6156845a11c4c95c7b44a8ac2911dd35db6e"}
Oct 30 22:30:03.289401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:03.289271    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" podStartSLOduration=3.28924422 podStartE2EDuration="3.28924422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.270427661 +0000 UTC m=+2414.794570447" watchObservedRunningTime="2026-10-30 22:30:03.28924422 +0000 UTC m=+2414.813386985"
Oct 30 22:30:03.764238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:03.760240    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:04.028888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.028769    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:04.103768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.103721    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerStarted","Data":"c8a6b0c649b3b481c69be532faaac86918145dc05a08d356f95c7ae6ec9c4f22"}
Oct 30 22:30:04.165246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.158544    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"]
Oct 30 22:30:04.165246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.158822    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy" containerID="cri-o://ce9ac1a45096c29f78ff3efe7c89c9f3dc8d4ad8196c22b29551841ef1410d25" gracePeriod=30
Oct 30 22:30:04.208247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.203230    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerStarted","Data":"78b3ab6e15e03490b7258703b2233ca7cf7a30c031d80920cb70948d27526a6a"}
Oct 30 22:30:04.243232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.234825    5031 generic.go:356] "Generic (PLEG): container finished" podID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerID="0b083618239ab08d68f0dbd690fa6156845a11c4c95c7b44a8ac2911dd35db6e" exitCode=60
Oct 30 22:30:04.243232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.234912    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" event={"ID":"c32cfce6-24a9-4ed6-bece-ead7a13b5205","Type":"ContainerDied","Data":"0b083618239ab08d68f0dbd690fa6156845a11c4c95c7b44a8ac2911dd35db6e"}
Oct 30 22:30:04.243232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.234968    5031 scope.go:122] "RemoveContainer" containerID="c93418383e4ad18ad9c811336f6929aa79f68b32c453e7aaa5c51e071d714a97"
Oct 30 22:30:04.320073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.319960    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:04.344276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:04.343571    5031 scope.go:122] "RemoveContainer" containerID="0b083618239ab08d68f0dbd690fa6156845a11c4c95c7b44a8ac2911dd35db6e"
Oct 30 22:30:04.344276 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:04.343966    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6_odh-ai-gateway-infra(c32cfce6-24a9-4ed6-bece-ead7a13b5205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205"
Oct 30 22:30:05.246189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.246151    5031 generic.go:356] "Generic (PLEG): container finished" podID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerID="c8a6b0c649b3b481c69be532faaac86918145dc05a08d356f95c7ae6ec9c4f22" exitCode=60
Oct 30 22:30:05.246741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.246239    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerDied","Data":"c8a6b0c649b3b481c69be532faaac86918145dc05a08d356f95c7ae6ec9c4f22"}
Oct 30 22:30:05.246741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.246312    5031 scope.go:122] "RemoveContainer" containerID="366996d3466f3fbf6165701c1f5afd3e29600028b55b4d074b0db277856e91aa"
Oct 30 22:30:05.246741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.246681    5031 scope.go:122] "RemoveContainer" containerID="c8a6b0c649b3b481c69be532faaac86918145dc05a08d356f95c7ae6ec9c4f22"
Oct 30 22:30:05.247034 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:05.247011    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w_odh-ai-gateway-infra(ec330f8c-d455-48b0-a4e6-ec008d31d305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305"
Oct 30 22:30:05.248845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.248820    5031 generic.go:356] "Generic (PLEG): container finished" podID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerID="78b3ab6e15e03490b7258703b2233ca7cf7a30c031d80920cb70948d27526a6a" exitCode=60
Oct 30 22:30:05.249016 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.248988    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerDied","Data":"78b3ab6e15e03490b7258703b2233ca7cf7a30c031d80920cb70948d27526a6a"}
Oct 30 22:30:05.249170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.249153    5031 scope.go:122] "RemoveContainer" containerID="78b3ab6e15e03490b7258703b2233ca7cf7a30c031d80920cb70948d27526a6a"
Oct 30 22:30:05.249539 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:05.249519    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-gr96l_odh-ai-gateway-infra(66d2a1f3-15f5-497a-8f2a-68c9ff261f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a"
Oct 30 22:30:05.251348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.251327    5031 scope.go:122] "RemoveContainer" containerID="0b083618239ab08d68f0dbd690fa6156845a11c4c95c7b44a8ac2911dd35db6e"
Oct 30 22:30:05.251638 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:05.251613    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6_odh-ai-gateway-infra(c32cfce6-24a9-4ed6-bece-ead7a13b5205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205"
Oct 30 22:30:05.261323 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:05.261303    5031 scope.go:122] "RemoveContainer" containerID="84c68a7a7a20e9747044ace6929b8a83e4727a83103d484d5a764548eda8f22c"
Oct 30 22:30:06.000368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:06.000324    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:06.260992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:06.260905    5031 scope.go:122] "RemoveContainer" containerID="c8a6b0c649b3b481c69be532faaac86918145dc05a08d356f95c7ae6ec9c4f22"
Oct 30 22:30:06.261440 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:06.261249    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w_odh-ai-gateway-infra(ec330f8c-d455-48b0-a4e6-ec008d31d305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305"
Oct 30 22:30:06.262735 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:06.262716    5031 scope.go:122] "RemoveContainer" containerID="78b3ab6e15e03490b7258703b2233ca7cf7a30c031d80920cb70948d27526a6a"
Oct 30 22:30:06.262970 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:06.262953    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-gr96l_odh-ai-gateway-infra(66d2a1f3-15f5-497a-8f2a-68c9ff261f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a"
Oct 30 22:30:07.298246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:07.298162    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" podUID="e4d9c20c-be0d-4589-8695-5723c3737517" containerName="payload-processing" containerID="cri-o://69c91ad8ed84fce2d73e62eeb42665cb050c3756bb3f4648c20f6571f325083d" gracePeriod=25
Oct 30 22:30:09.289914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.289878    5031 generic.go:356] "Generic (PLEG): container finished" podID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerID="ce9ac1a45096c29f78ff3efe7c89c9f3dc8d4ad8196c22b29551841ef1410d25" exitCode=0
Oct 30 22:30:09.290369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.289958    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" event={"ID":"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014","Type":"ContainerDied","Data":"ce9ac1a45096c29f78ff3efe7c89c9f3dc8d4ad8196c22b29551841ef1410d25"}
Oct 30 22:30:09.290369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.290040    5031 scope.go:122] "RemoveContainer" containerID="ff5c4f8d7cddf333b84b76d79bc77d9fdfcc44f69356970f8943dfcfff997581"
Oct 30 22:30:09.411852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.411811    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:30:09.446666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446583    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-kube-api-access-lnr7g\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-kube-api-access-lnr7g\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.446666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446652    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-data\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.446903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446684    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-credential-socket\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.446903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446718    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-certs\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.446903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446790    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-token\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-token\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.446903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446855    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-envoy\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.447125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446933    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istiod-ca-cert\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.447125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446954    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-data" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:09.447125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446974    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-socket\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.447125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.446994    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-credential-socket" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:09.447125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447010    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-podinfo\") pod \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\" (UID: \"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014\") "
Oct 30 22:30:09.447125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447048    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-certs" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:09.447501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447234    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-socket" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:09.447501 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447344    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istiod-ca-cert" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:09.447672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447560    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.447672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447583    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.447672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447600    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.447672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447615    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.447672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.447628    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.449403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.449378    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-envoy" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:09.449562 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.449545    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-token" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:09.449867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.449843    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-podinfo" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:09.449867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.449847    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-kube-api-access-lnr7g" pod "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" (UID: "f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014"). InnerVolumeSpecName "kube-api-access-lnr7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:09.549025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.548989    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.549025 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.549018    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.549239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.549039    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:09.549239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:09.549050    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnr7g\" (UniqueName: \"kubernetes.io/projected/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014-kube-api-access-lnr7g\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.299705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.299672    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"
Oct 30 22:30:10.300259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.299708    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc" event={"ID":"f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014","Type":"ContainerDied","Data":"9b1ac52d578e9204f421ef68fd1102ecb4789f3606924c1165d0a7c6fc73efd6"}
Oct 30 22:30:10.300259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.299765    5031 scope.go:122] "RemoveContainer" containerID="ce9ac1a45096c29f78ff3efe7c89c9f3dc8d4ad8196c22b29551841ef1410d25"
Oct 30 22:30:10.472758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.472721    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"]
Oct 30 22:30:10.480383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.480355    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc"]
Oct 30 22:30:10.647010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.646971    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" path="/var/lib/kubelet/pods/f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014/volumes"
Oct 30 22:30:10.653297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.653264    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"]
Oct 30 22:30:10.653923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.653903    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy"
Oct 30 22:30:10.654040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.653924    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy"
Oct 30 22:30:10.654040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.653938    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy"
Oct 30 22:30:10.654040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.653946    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy"
Oct 30 22:30:10.654231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.654120    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy"
Oct 30 22:30:10.654231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.654136    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d4ccaf-f1e8-4aae-8351-98ff0b3f9014" containerName="istio-proxy"
Oct 30 22:30:10.658807 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.658785    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.713682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.713652    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:10.716394 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.716353    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"]
Oct 30 22:30:10.763939 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.763907    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.764136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.763995    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.764202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.764164    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcwcq\" (UniqueName: \"kubernetes.io/projected/0da3cdce-0288-42ed-9579-e8c239783ba3-kube-api-access-rcwcq\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.865537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.865495    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcwcq\" (UniqueName: \"kubernetes.io/projected/0da3cdce-0288-42ed-9579-e8c239783ba3-kube-api-access-rcwcq\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.865713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.865611    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.865713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.865663    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.866485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.866465    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.868361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.868334    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.932769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.932694    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcwcq\" (UniqueName: \"kubernetes.io/projected/0da3cdce-0288-42ed-9579-e8c239783ba3-kube-api-access-rcwcq\") pod \"maas-api-revoke-sub-2a03aa0f-krnx5\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:10.973494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:10.973444    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:11.343570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.343536    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" containerID="cri-o://3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e" gracePeriod=45
Oct 30 22:30:11.356586 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.356553    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"]
Oct 30 22:30:11.361819 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:30:11.361789    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da3cdce_0288_42ed_9579_e8c239783ba3.slice/crio-eb2269c08422fdd419eceab9b49a4376f09e447a21648462e0400a18c0ec1e06 WatchSource:0}: Error finding container eb2269c08422fdd419eceab9b49a4376f09e447a21648462e0400a18c0ec1e06: Status 404 returned error can't find the container with id eb2269c08422fdd419eceab9b49a4376f09e447a21648462e0400a18c0ec1e06
Oct 30 22:30:11.411021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.410964    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": dial tcp 10.132.0.107:8000: connect: connection refused"
Oct 30 22:30:11.604818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.604729    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.629002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.628972    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:30:11.677815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.677780    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-home\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-home\") pod \"a9f65885-52f6-4462-90cf-dd806344c29a\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") "
Oct 30 22:30:11.678051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.677871    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-model-cache\") pod \"a9f65885-52f6-4462-90cf-dd806344c29a\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") "
Oct 30 22:30:11.678051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.677912    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f65885-52f6-4462-90cf-dd806344c29a-kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/a9f65885-52f6-4462-90cf-dd806344c29a-kube-api-access-7qhf2\") pod \"a9f65885-52f6-4462-90cf-dd806344c29a\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") "
Oct 30 22:30:11.678051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.677998    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-tmp-dir\") pod \"a9f65885-52f6-4462-90cf-dd806344c29a\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") "
Oct 30 22:30:11.678051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678043    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-dshm\") pod \"a9f65885-52f6-4462-90cf-dd806344c29a\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") "
Oct 30 22:30:11.678289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678074    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f65885-52f6-4462-90cf-dd806344c29a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f65885-52f6-4462-90cf-dd806344c29a-tls-certs\") pod \"a9f65885-52f6-4462-90cf-dd806344c29a\" (UID: \"a9f65885-52f6-4462-90cf-dd806344c29a\") "
Oct 30 22:30:11.678289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678148    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-home" pod "a9f65885-52f6-4462-90cf-dd806344c29a" (UID: "a9f65885-52f6-4462-90cf-dd806344c29a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.678289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678202    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-model-cache" pod "a9f65885-52f6-4462-90cf-dd806344c29a" (UID: "a9f65885-52f6-4462-90cf-dd806344c29a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.678446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678383    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-tmp-dir" pod "a9f65885-52f6-4462-90cf-dd806344c29a" (UID: "a9f65885-52f6-4462-90cf-dd806344c29a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.678751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678599    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.678751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678625    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.678751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.678659    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.681102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.681067    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-dshm" pod "a9f65885-52f6-4462-90cf-dd806344c29a" (UID: "a9f65885-52f6-4462-90cf-dd806344c29a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:11.681102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.681092    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f65885-52f6-4462-90cf-dd806344c29a-kube-api-access-7qhf2" pod "a9f65885-52f6-4462-90cf-dd806344c29a" (UID: "a9f65885-52f6-4462-90cf-dd806344c29a"). InnerVolumeSpecName "kube-api-access-7qhf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:11.681287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.681162    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f65885-52f6-4462-90cf-dd806344c29a-tls-certs" pod "a9f65885-52f6-4462-90cf-dd806344c29a" (UID: "a9f65885-52f6-4462-90cf-dd806344c29a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:11.780287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.780245    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f65885-52f6-4462-90cf-dd806344c29a-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.780287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.780283    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f65885-52f6-4462-90cf-dd806344c29a-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.780513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:11.780296    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/a9f65885-52f6-4462-90cf-dd806344c29a-kube-api-access-7qhf2\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:12.213825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.213728    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" podUID="e4d9c20c-be0d-4589-8695-5723c3737517" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:12.317920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.317869    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5" event={"ID":"0da3cdce-0288-42ed-9579-e8c239783ba3","Type":"ContainerStarted","Data":"55db83119facd6cb773cfecba2479088ec2c1c00770a915f6066698883255254"}
Oct 30 22:30:12.318141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.317934    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5" event={"ID":"0da3cdce-0288-42ed-9579-e8c239783ba3","Type":"ContainerStarted","Data":"eb2269c08422fdd419eceab9b49a4376f09e447a21648462e0400a18c0ec1e06"}
Oct 30 22:30:12.319301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.319272    5031 generic.go:356] "Generic (PLEG): container finished" podID="a9f65885-52f6-4462-90cf-dd806344c29a" containerID="3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e" exitCode=0
Oct 30 22:30:12.319426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.319309    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"
Oct 30 22:30:12.319426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.319341    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" event={"ID":"a9f65885-52f6-4462-90cf-dd806344c29a","Type":"ContainerDied","Data":"3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e"}
Oct 30 22:30:12.319426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.319379    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27" event={"ID":"a9f65885-52f6-4462-90cf-dd806344c29a","Type":"ContainerDied","Data":"d990d9e714d0a8c479556a1c8cf8a89c54d9b24830ac948538a0f51bfeab62b1"}
Oct 30 22:30:12.319426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.319395    5031 scope.go:122] "RemoveContainer" containerID="3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e"
Oct 30 22:30:12.331230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.331191    5031 scope.go:122] "RemoveContainer" containerID="3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e"
Oct 30 22:30:12.331555 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:12.331534    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e\": container with ID starting with 3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e not found: ID does not exist" containerID="3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e"
Oct 30 22:30:12.331619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.331563    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e"} err="failed to get container status \"3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e\": rpc error: code = NotFound desc = could not find container \"3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e\": container with ID starting with 3732b0f95d0152f2d47d1ec41f0c291a1670abb4108f298a15c8f5fd7d07053e not found: ID does not exist"
Oct 30 22:30:12.438394 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.438341    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5" podStartSLOduration=2.438321519 podStartE2EDuration="2.438321519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:12.437414113 +0000 UTC m=+2423.961556878" watchObservedRunningTime="2026-10-30 22:30:12.438321519 +0000 UTC m=+2423.962464283"
Oct 30 22:30:12.554925 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.554832    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"]
Oct 30 22:30:12.558296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.558268    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/distinct-w5-3783c2-kserve-79dd8c88b6-lkx27"]
Oct 30 22:30:12.646201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:12.646165    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" path="/var/lib/kubelet/pods/a9f65885-52f6-4462-90cf-dd806344c29a/volumes"
Oct 30 22:30:13.060522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:13.060493    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"]
Oct 30 22:30:13.064076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:13.064048    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd38b61-vzwnn"]
Oct 30 22:30:13.067152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:13.067126    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"]
Oct 30 22:30:13.110570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:13.110539    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e78729-fcpwk"]
Oct 30 22:30:13.328514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:13.328420    5031 generic.go:356] "Generic (PLEG): container finished" podID="0da3cdce-0288-42ed-9579-e8c239783ba3" containerID="55db83119facd6cb773cfecba2479088ec2c1c00770a915f6066698883255254" exitCode=0
Oct 30 22:30:13.328707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:13.328507    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5" event={"ID":"0da3cdce-0288-42ed-9579-e8c239783ba3","Type":"ContainerDied","Data":"55db83119facd6cb773cfecba2479088ec2c1c00770a915f6066698883255254"}
Oct 30 22:30:13.639044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:13.639013    5031 scope.go:122] "RemoveContainer" containerID="0b083618239ab08d68f0dbd690fa6156845a11c4c95c7b44a8ac2911dd35db6e"
Oct 30 22:30:14.341551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.341456    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" event={"ID":"c32cfce6-24a9-4ed6-bece-ead7a13b5205","Type":"ContainerStarted","Data":"b49b970c490255da8b32b66d6e4584b18bd5d9ac6e4c535d243d0b75c8aef84b"}
Oct 30 22:30:14.505991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.505957    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:14.608770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.608731    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-xks-ca\") pod \"0da3cdce-0288-42ed-9579-e8c239783ba3\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") "
Oct 30 22:30:14.608981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.608867    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da3cdce-0288-42ed-9579-e8c239783ba3-kube-api-access-rcwcq\" (UniqueName: \"kubernetes.io/projected/0da3cdce-0288-42ed-9579-e8c239783ba3-kube-api-access-rcwcq\") pod \"0da3cdce-0288-42ed-9579-e8c239783ba3\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") "
Oct 30 22:30:14.608981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.608940    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-openshift-ca\") pod \"0da3cdce-0288-42ed-9579-e8c239783ba3\" (UID: \"0da3cdce-0288-42ed-9579-e8c239783ba3\") "
Oct 30 22:30:14.609383 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.609356    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-openshift-ca" pod "0da3cdce-0288-42ed-9579-e8c239783ba3" (UID: "0da3cdce-0288-42ed-9579-e8c239783ba3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:14.611221 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.611187    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-xks-ca" pod "0da3cdce-0288-42ed-9579-e8c239783ba3" (UID: "0da3cdce-0288-42ed-9579-e8c239783ba3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:14.611319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.611231    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da3cdce-0288-42ed-9579-e8c239783ba3-kube-api-access-rcwcq" pod "0da3cdce-0288-42ed-9579-e8c239783ba3" (UID: "0da3cdce-0288-42ed-9579-e8c239783ba3"). InnerVolumeSpecName "kube-api-access-rcwcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.649658 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.649617    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414303a8-eacb-4472-8115-f8be154c96d1" path="/var/lib/kubelet/pods/414303a8-eacb-4472-8115-f8be154c96d1/volumes"
Oct 30 22:30:14.650300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.650283    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb469053-f5e2-4e90-9f98-be803e987d25" path="/var/lib/kubelet/pods/eb469053-f5e2-4e90-9f98-be803e987d25/volumes"
Oct 30 22:30:14.651548 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.651517    5031 scope.go:122] "RemoveContainer" containerID="c8a6b0c649b3b481c69be532faaac86918145dc05a08d356f95c7ae6ec9c4f22"
Oct 30 22:30:14.710073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.710035    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:14.710073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.710074    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da3cdce-0288-42ed-9579-e8c239783ba3-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:14.710331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:14.710092    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcwcq\" (UniqueName: \"kubernetes.io/projected/0da3cdce-0288-42ed-9579-e8c239783ba3-kube-api-access-rcwcq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:15.352287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.352245    5031 generic.go:356] "Generic (PLEG): container finished" podID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerID="b49b970c490255da8b32b66d6e4584b18bd5d9ac6e4c535d243d0b75c8aef84b" exitCode=60
Oct 30 22:30:15.352487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.352290    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" event={"ID":"c32cfce6-24a9-4ed6-bece-ead7a13b5205","Type":"ContainerDied","Data":"b49b970c490255da8b32b66d6e4584b18bd5d9ac6e4c535d243d0b75c8aef84b"}
Oct 30 22:30:15.352487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.352383    5031 scope.go:122] "RemoveContainer" containerID="0b083618239ab08d68f0dbd690fa6156845a11c4c95c7b44a8ac2911dd35db6e"
Oct 30 22:30:15.352769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.352747    5031 scope.go:122] "RemoveContainer" containerID="b49b970c490255da8b32b66d6e4584b18bd5d9ac6e4c535d243d0b75c8aef84b"
Oct 30 22:30:15.353121 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:15.353094    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6_odh-ai-gateway-infra(c32cfce6-24a9-4ed6-bece-ead7a13b5205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205"
Oct 30 22:30:15.354132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.354112    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"
Oct 30 22:30:15.354285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.354149    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5" event={"ID":"0da3cdce-0288-42ed-9579-e8c239783ba3","Type":"ContainerDied","Data":"eb2269c08422fdd419eceab9b49a4376f09e447a21648462e0400a18c0ec1e06"}
Oct 30 22:30:15.354285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.354195    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb2269c08422fdd419eceab9b49a4376f09e447a21648462e0400a18c0ec1e06"
Oct 30 22:30:15.356714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.356671    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerStarted","Data":"cb57a1876d20f9aa3b38e6f8c58493bd0c9eb303a5cfdb7c5a322bc0006290ef"}
Oct 30 22:30:15.505257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.505222    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"]
Oct 30 22:30:15.638283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.638253    5031 scope.go:122] "RemoveContainer" containerID="78b3ab6e15e03490b7258703b2233ca7cf7a30c031d80920cb70948d27526a6a"
Oct 30 22:30:15.995519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:15.995484    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:16.227374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.227290    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:16.368054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.368008    5031 generic.go:356] "Generic (PLEG): container finished" podID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerID="cb57a1876d20f9aa3b38e6f8c58493bd0c9eb303a5cfdb7c5a322bc0006290ef" exitCode=60
Oct 30 22:30:16.368177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.368079    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerDied","Data":"cb57a1876d20f9aa3b38e6f8c58493bd0c9eb303a5cfdb7c5a322bc0006290ef"}
Oct 30 22:30:16.368177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.368143    5031 scope.go:122] "RemoveContainer" containerID="c8a6b0c649b3b481c69be532faaac86918145dc05a08d356f95c7ae6ec9c4f22"
Oct 30 22:30:16.368653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.368637    5031 scope.go:122] "RemoveContainer" containerID="cb57a1876d20f9aa3b38e6f8c58493bd0c9eb303a5cfdb7c5a322bc0006290ef"
Oct 30 22:30:16.368990 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:16.368964    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w_odh-ai-gateway-infra(ec330f8c-d455-48b0-a4e6-ec008d31d305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305"
Oct 30 22:30:16.370375 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.370354    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerStarted","Data":"13bc675f8fc25a5dcef80cffffffa3c4cb49e0ba7b02ba44843ef827044c620e"}
Oct 30 22:30:16.581992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.581965    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:16.629338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.629302    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32cfce6-24a9-4ed6-bece-ead7a13b5205-kube-api-access-8sssp\" (UniqueName: \"kubernetes.io/projected/c32cfce6-24a9-4ed6-bece-ead7a13b5205-kube-api-access-8sssp\") pod \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") "
Oct 30 22:30:16.629569 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.629421    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32cfce6-24a9-4ed6-bece-ead7a13b5205-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32cfce6-24a9-4ed6-bece-ead7a13b5205-maas-api-service-ca\") pod \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\" (UID: \"c32cfce6-24a9-4ed6-bece-ead7a13b5205\") "
Oct 30 22:30:16.629859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.629830    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32cfce6-24a9-4ed6-bece-ead7a13b5205-maas-api-service-ca" pod "c32cfce6-24a9-4ed6-bece-ead7a13b5205" (UID: "c32cfce6-24a9-4ed6-bece-ead7a13b5205"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:16.632355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.632324    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32cfce6-24a9-4ed6-bece-ead7a13b5205-kube-api-access-8sssp" pod "c32cfce6-24a9-4ed6-bece-ead7a13b5205" (UID: "c32cfce6-24a9-4ed6-bece-ead7a13b5205"). InnerVolumeSpecName "kube-api-access-8sssp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:16.681260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.681219    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"]
Oct 30 22:30:16.765298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.753027    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sssp\" (UniqueName: \"kubernetes.io/projected/c32cfce6-24a9-4ed6-bece-ead7a13b5205-kube-api-access-8sssp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.765298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.753055    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32cfce6-24a9-4ed6-bece-ead7a13b5205-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.865233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:16.855734    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:17.092037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.091939    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:17.386063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.385967    5031 generic.go:356] "Generic (PLEG): container finished" podID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerID="13bc675f8fc25a5dcef80cffffffa3c4cb49e0ba7b02ba44843ef827044c620e" exitCode=60
Oct 30 22:30:17.386063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.386043    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerDied","Data":"13bc675f8fc25a5dcef80cffffffa3c4cb49e0ba7b02ba44843ef827044c620e"}
Oct 30 22:30:17.386356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.386123    5031 scope.go:122] "RemoveContainer" containerID="78b3ab6e15e03490b7258703b2233ca7cf7a30c031d80920cb70948d27526a6a"
Oct 30 22:30:17.386594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.386577    5031 scope.go:122] "RemoveContainer" containerID="13bc675f8fc25a5dcef80cffffffa3c4cb49e0ba7b02ba44843ef827044c620e"
Oct 30 22:30:17.386876 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:17.386854    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-gr96l_odh-ai-gateway-infra(66d2a1f3-15f5-497a-8f2a-68c9ff261f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a"
Oct 30 22:30:17.388339 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.388319    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"
Oct 30 22:30:17.388448 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.388322    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6" event={"ID":"c32cfce6-24a9-4ed6-bece-ead7a13b5205","Type":"ContainerDied","Data":"20b3ddb95b7a532a78e901cf98e10f7e858897fd8b7c0b94ff9cfe52adb5f593"}
Oct 30 22:30:17.421712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.421676    5031 scope.go:122] "RemoveContainer" containerID="b49b970c490255da8b32b66d6e4584b18bd5d9ac6e4c535d243d0b75c8aef84b"
Oct 30 22:30:17.497640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.497605    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:17.654552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.653549    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:17.663777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.663745    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"]
Oct 30 22:30:17.670428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.670401    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d59c58-29889990-q7tz6"]
Oct 30 22:30:17.722297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.722268    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:17.745960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.745928    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"]
Oct 30 22:30:17.770071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.770042    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec330f8c-d455-48b0-a4e6-ec008d31d305-kube-api-access-cs7jf\" (UniqueName: \"kubernetes.io/projected/ec330f8c-d455-48b0-a4e6-ec008d31d305-kube-api-access-cs7jf\") pod \"ec330f8c-d455-48b0-a4e6-ec008d31d305\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") "
Oct 30 22:30:17.770249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.770166    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec330f8c-d455-48b0-a4e6-ec008d31d305-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec330f8c-d455-48b0-a4e6-ec008d31d305-maas-api-service-ca\") pod \"ec330f8c-d455-48b0-a4e6-ec008d31d305\" (UID: \"ec330f8c-d455-48b0-a4e6-ec008d31d305\") "
Oct 30 22:30:17.770565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.770533    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec330f8c-d455-48b0-a4e6-ec008d31d305-maas-api-service-ca" pod "ec330f8c-d455-48b0-a4e6-ec008d31d305" (UID: "ec330f8c-d455-48b0-a4e6-ec008d31d305"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.772435 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.772414    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec330f8c-d455-48b0-a4e6-ec008d31d305-kube-api-access-cs7jf" pod "ec330f8c-d455-48b0-a4e6-ec008d31d305" (UID: "ec330f8c-d455-48b0-a4e6-ec008d31d305"). InnerVolumeSpecName "kube-api-access-cs7jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.872048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.872013    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7jf\" (UniqueName: \"kubernetes.io/projected/ec330f8c-d455-48b0-a4e6-ec008d31d305-kube-api-access-cs7jf\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.872048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:17.872041    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec330f8c-d455-48b0-a4e6-ec008d31d305-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.401529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.401488    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"
Oct 30 22:30:18.401529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.401503    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w" event={"ID":"ec330f8c-d455-48b0-a4e6-ec008d31d305","Type":"ContainerDied","Data":"4134c9835ff1c92c9d6f1ff77ae80e55ddec99ed97f6823be45c0b9f58d277b9"}
Oct 30 22:30:18.402087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.401578    5031 scope.go:122] "RemoveContainer" containerID="cb57a1876d20f9aa3b38e6f8c58493bd0c9eb303a5cfdb7c5a322bc0006290ef"
Oct 30 22:30:18.547109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.547071    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:18.573605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.573569    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"]
Oct 30 22:30:18.625708 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.625668    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8ca52a-29889990-lfm5w"]
Oct 30 22:30:18.649050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.649016    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" path="/var/lib/kubelet/pods/c32cfce6-24a9-4ed6-bece-ead7a13b5205/volumes"
Oct 30 22:30:18.649697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.649676    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" path="/var/lib/kubelet/pods/ec330f8c-d455-48b0-a4e6-ec008d31d305/volumes"
Oct 30 22:30:18.683968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.683862    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-maas-api-service-ca\") pod \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") "
Oct 30 22:30:18.684153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.684026    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-kube-api-access-5mssr\" (UniqueName: \"kubernetes.io/projected/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-kube-api-access-5mssr\") pod \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\" (UID: \"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a\") "
Oct 30 22:30:18.684469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.684437    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-maas-api-service-ca" pod "66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" (UID: "66d2a1f3-15f5-497a-8f2a-68c9ff261f2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.684610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.684594    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.687689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.687657    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-kube-api-access-5mssr" pod "66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" (UID: "66d2a1f3-15f5-497a-8f2a-68c9ff261f2a"). InnerVolumeSpecName "kube-api-access-5mssr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.733412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.733371    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:30:18.786343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:18.786300    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mssr\" (UniqueName: \"kubernetes.io/projected/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a-kube-api-access-5mssr\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.420348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:19.419202    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"
Oct 30 22:30:19.421591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:19.421547    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l" event={"ID":"66d2a1f3-15f5-497a-8f2a-68c9ff261f2a","Type":"ContainerDied","Data":"b84e757f85b77686475819fb7479f0b6b1585535a723ca0c45fc000dda684023"}
Oct 30 22:30:19.421750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:19.421635    5031 scope.go:122] "RemoveContainer" containerID="13bc675f8fc25a5dcef80cffffffa3c4cb49e0ba7b02ba44843ef827044c620e"
Oct 30 22:30:19.594344 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:19.594301    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"]
Oct 30 22:30:19.646627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:19.646589    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-gr96l"]
Oct 30 22:30:20.544404 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:20.544366    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:30:20.645719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:20.645688    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" path="/var/lib/kubelet/pods/66d2a1f3-15f5-497a-8f2a-68c9ff261f2a/volumes"
Oct 30 22:30:20.921831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:20.921799    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:22.059694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:22.059655    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:22.214135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:22.214094    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" podUID="e4d9c20c-be0d-4589-8695-5723c3737517" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:22.318389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:22.318305    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:26.670541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:26.670486    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:27.534653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.534618    5031 generic.go:356] "Generic (PLEG): container finished" podID="e4d9c20c-be0d-4589-8695-5723c3737517" containerID="69c91ad8ed84fce2d73e62eeb42665cb050c3756bb3f4648c20f6571f325083d" exitCode=0
Oct 30 22:30:27.534892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.534698    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" event={"ID":"e4d9c20c-be0d-4589-8695-5723c3737517","Type":"ContainerDied","Data":"69c91ad8ed84fce2d73e62eeb42665cb050c3756bb3f4648c20f6571f325083d"}
Oct 30 22:30:27.620044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.620015    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 22:30:27.623133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.623113    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d9c20c-be0d-4589-8695-5723c3737517-kube-api-access-t6x7h\" (UniqueName: \"kubernetes.io/projected/e4d9c20c-be0d-4589-8695-5723c3737517-kube-api-access-t6x7h\") pod \"e4d9c20c-be0d-4589-8695-5723c3737517\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") "
Oct 30 22:30:27.623303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.623286    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d9c20c-be0d-4589-8695-5723c3737517-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d9c20c-be0d-4589-8695-5723c3737517-plugins-config-volume\") pod \"e4d9c20c-be0d-4589-8695-5723c3737517\" (UID: \"e4d9c20c-be0d-4589-8695-5723c3737517\") "
Oct 30 22:30:27.623637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.623613    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d9c20c-be0d-4589-8695-5723c3737517-plugins-config-volume" pod "e4d9c20c-be0d-4589-8695-5723c3737517" (UID: "e4d9c20c-be0d-4589-8695-5723c3737517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:27.625479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.625457    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d9c20c-be0d-4589-8695-5723c3737517-kube-api-access-t6x7h" pod "e4d9c20c-be0d-4589-8695-5723c3737517" (UID: "e4d9c20c-be0d-4589-8695-5723c3737517"). InnerVolumeSpecName "kube-api-access-t6x7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:27.724485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.724449    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d9c20c-be0d-4589-8695-5723c3737517-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:27.724485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:27.724477    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6x7h\" (UniqueName: \"kubernetes.io/projected/e4d9c20c-be0d-4589-8695-5723c3737517-kube-api-access-t6x7h\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:28.504623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.504558    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:28.548020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.546070    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g" event={"ID":"e4d9c20c-be0d-4589-8695-5723c3737517","Type":"ContainerDied","Data":"f94209dbc0c86daf5a71b428728a690d8c04bb11d9f5cbd467f2e5f88c8c1980"}
Oct 30 22:30:28.548020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.546143    5031 scope.go:122] "RemoveContainer" containerID="69c91ad8ed84fce2d73e62eeb42665cb050c3756bb3f4648c20f6571f325083d"
Oct 30 22:30:28.548020 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.546345    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"
Oct 30 22:30:28.728754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.728716    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"]
Oct 30 22:30:28.758091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.758007    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:28.779526 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.779495    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g"]
Oct 30 22:30:28.814602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.814564    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"]
Oct 30 22:30:28.818983 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:28.818957    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2182a5-6rrck"]
Oct 30 22:30:29.005132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:29.005100    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:29.282779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:29.282743    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:30.419035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:30.418993    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:30.645626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:30.645587    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d9c20c-be0d-4589-8695-5723c3737517" path="/var/lib/kubelet/pods/e4d9c20c-be0d-4589-8695-5723c3737517/volumes"
Oct 30 22:30:30.646081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:30.646067    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66a45c5-1eca-4ea8-80b9-0746bf069ffc" path="/var/lib/kubelet/pods/e66a45c5-1eca-4ea8-80b9-0746bf069ffc/volumes"
Oct 30 22:30:33.451353 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:33.451314    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:34.644541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:34.644507    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:35.099035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:35.098996    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:35.345840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:35.345802    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:35.905552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:35.905517    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.140969 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:36.140932    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.376196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:36.376163    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.605559 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:36.605527    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.833988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:36.833890    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"]
Oct 30 22:30:36.884163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:36.884126    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3783c2-b5zzf"]
Oct 30 22:30:36.939260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:36.939224    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:37.490853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:37.490815    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:37.744288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:37.744193    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:37.968360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:37.968328    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:38.645308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:38.645273    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5112b3ef-d243-47d4-8d3b-c6921f1a624b" path="/var/lib/kubelet/pods/5112b3ef-d243-47d4-8d3b-c6921f1a624b/volumes"
Oct 30 22:30:41.916936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:41.916897    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:44.531505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:44.531465    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:44.762673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:44.762618    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:44.996105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:44.996072    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:45.360168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:45.360134    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.266231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:46.266175    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.451269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:46.451235    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"]
Oct 30 22:30:46.451557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:46.451460    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy" containerID="cri-o://f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c" gracePeriod=30
Oct 30 22:30:46.483991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:46.483948    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.712872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:46.712833    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.254460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:47.254412    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:48.981172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:48.981135    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.220426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:49.220380    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:50.250859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:50.250815    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"]
Oct 30 22:30:50.433433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:50.433396    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:50.541976 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:50.541886    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:30:51.731769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.731693    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 22:30:51.736236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.736189    5031 generic.go:356] "Generic (PLEG): container finished" podID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerID="f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c" exitCode=0
Oct 30 22:30:51.736388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.736245    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"
Oct 30 22:30:51.736388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.736256    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" event={"ID":"646d59da-3bfd-4e6a-a6b1-5e387db1bb44","Type":"ContainerDied","Data":"f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c"}
Oct 30 22:30:51.736388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.736314    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6" event={"ID":"646d59da-3bfd-4e6a-a6b1-5e387db1bb44","Type":"ContainerDied","Data":"cd1827c2db5f45c8012cebacf587eb45d44a78f8a7b717fe51d9c746321b28fb"}
Oct 30 22:30:51.736388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.736345    5031 scope.go:122] "RemoveContainer" containerID="f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c"
Oct 30 22:30:51.752102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.752081    5031 scope.go:122] "RemoveContainer" containerID="4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"
Oct 30 22:30:51.762672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.762637    5031 scope.go:122] "RemoveContainer" containerID="aa6d22555e00d33f6c1196289ebd542ff607bc686e5839c266f571e17367c041"
Oct 30 22:30:51.764189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.764149    5031 scope.go:122] "RemoveContainer" containerID="f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c"
Oct 30 22:30:51.764669 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:51.764638    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c\": container with ID starting with f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c not found: ID does not exist" containerID="f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c"
Oct 30 22:30:51.764799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.764682    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c"} err="failed to get container status \"f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c\": rpc error: code = NotFound desc = could not find container \"f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c\": container with ID starting with f71436f83fe7bf9ef1d5e3aaf9dd1c97f0f2c2508fb57bba8c5e39f85361539c not found: ID does not exist"
Oct 30 22:30:51.764799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.764713    5031 scope.go:122] "RemoveContainer" containerID="4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"
Oct 30 22:30:51.766482 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:51.766450    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3\": container with ID starting with 4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3 not found: ID does not exist" containerID="4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"
Oct 30 22:30:51.766591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.766491    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"} err="failed to get container status \"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3\": rpc error: code = NotFound desc = could not find container \"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3\": container with ID starting with 4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3 not found: ID does not exist"
Oct 30 22:30:51.773246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.773226    5031 scope.go:122] "RemoveContainer" containerID="fe9978f66614a11eda1f7891241003e23acc2dd0ff6f55b9ac11dc8614c580c2"
Oct 30 22:30:51.782911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.782879    5031 scope.go:122] "RemoveContainer" containerID="4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"
Oct 30 22:30:51.783200 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:30:51.783178    5031 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3\": rpc error: code = NotFound desc = could not find container \"4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3\": container with ID starting with 4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3 not found: ID does not exist" containerID="4c8cba07b8c37d6c4ecae86d4914c278a732a04c5055c006476420a0f0eab4e3"
Oct 30 22:30:51.783322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.783224    5031 scope.go:122] "RemoveContainer" containerID="342663b47b7bdc36b2b22dacb388d1a5b010f0ec93c75a68ca0c71d8bd885e45"
Oct 30 22:30:51.792793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.792770    5031 scope.go:122] "RemoveContainer" containerID="4daf85a01e90f53500e578f2f45dab5fb3af1b335b274914f2a42c4f061d8ce5"
Oct 30 22:30:51.818492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818463    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-token\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-token\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.818617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818534    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-data\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.818681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818620    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-socket\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.818681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818666    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-podinfo\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.818805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818733    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istiod-ca-cert\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.818805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818773    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-envoy\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.818911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818834    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-certs\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.818911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818883    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-credential-socket\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.819030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818901    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-data" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.819030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818919    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-kube-api-access-m5rdz\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-kube-api-access-m5rdz\") pod \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\" (UID: \"646d59da-3bfd-4e6a-a6b1-5e387db1bb44\") "
Oct 30 22:30:51.819030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.818917    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-socket" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.819187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819163    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-certs" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.819512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819475    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-credential-socket" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.819625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819504    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istiod-ca-cert" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:51.819811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819769    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.819811 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819800    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.819979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819823    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.819979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819839    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.819979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.819853    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.821396 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.821372    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-envoy" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:51.821520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.821502    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-kube-api-access-m5rdz" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "kube-api-access-m5rdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:51.821588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.821525    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-podinfo" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:51.821843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.821828    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-token" pod "646d59da-3bfd-4e6a-a6b1-5e387db1bb44" (UID: "646d59da-3bfd-4e6a-a6b1-5e387db1bb44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:51.920954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.920917    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5rdz\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-kube-api-access-m5rdz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.921094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.920958    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.921094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.920973    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.921094 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:51.920987    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d59da-3bfd-4e6a-a6b1-5e387db1bb44-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:52.162731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:52.162695    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"]
Oct 30 22:30:52.169579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:52.169541    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6"]
Oct 30 22:30:52.616141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:52.616104    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:52.645686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:52.645656    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" path="/var/lib/kubelet/pods/646d59da-3bfd-4e6a-a6b1-5e387db1bb44/volumes"
Oct 30 22:30:53.232574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:53.232537    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.461462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:53.461426    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:56.204482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:56.204437    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"]
Oct 30 22:30:56.314742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:56.314704    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"]
Oct 30 22:30:56.661712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:56.661666    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:56.917396 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:56.917301    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:57.185590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.185506    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"]
Oct 30 22:30:57.186070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186056    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:30:57.186122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186072    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:30:57.186122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186084    5031 cpu_manager.go:414] "containerMap: removing container" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy"
Oct 30 22:30:57.186122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186089    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy"
Oct 30 22:30:57.186122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186102    5031 cpu_manager.go:414] "containerMap: removing container" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy"
Oct 30 22:30:57.186122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186108    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy"
Oct 30 22:30:57.186122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186116    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e4d9c20c-be0d-4589-8695-5723c3737517" containerName="payload-processing"
Oct 30 22:30:57.186122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186122    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d9c20c-be0d-4589-8695-5723c3737517" containerName="payload-processing"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186129    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186134    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186144    5031 cpu_manager.go:414] "containerMap: removing container" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186149    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186156    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0da3cdce-0288-42ed-9579-e8c239783ba3" containerName="revoke-keys"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186160    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da3cdce-0288-42ed-9579-e8c239783ba3" containerName="revoke-keys"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186166    5031 cpu_manager.go:414] "containerMap: removing container" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186170    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186177    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186181    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186191    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186195    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186229    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186235    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186315    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186323    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da3cdce-0288-42ed-9579-e8c239783ba3" containerName="revoke-keys"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186328    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186337    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f65885-52f6-4462-90cf-dd806344c29a" containerName="main"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186343    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186349    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186354    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186359    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186365    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186371    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186376    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="646d59da-3bfd-4e6a-a6b1-5e387db1bb44" containerName="istio-proxy"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186382    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d9c20c-be0d-4589-8695-5723c3737517" containerName="payload-processing"
Oct 30 22:30:57.186374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.186388    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:30:57.192201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.192180    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.246690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.246660    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:57.247176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.246758    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:30:57.247176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.246993    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:57.247176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.247072    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:57.249600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.249571    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"]
Oct 30 22:30:57.379623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.379540    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.379623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.379603    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswmb\" (UniqueName: \"kubernetes.io/projected/8a31206a-fa53-4e11-86eb-f891ab1b2d11-kube-api-access-cswmb\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.379891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.379742    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.480447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.480359    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswmb\" (UniqueName: \"kubernetes.io/projected/8a31206a-fa53-4e11-86eb-f891ab1b2d11-kube-api-access-cswmb\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.480901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.480442    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.480901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.480548    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.481271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.481251    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.483087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.483068    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.543621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.543584    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswmb\" (UniqueName: \"kubernetes.io/projected/8a31206a-fa53-4e11-86eb-f891ab1b2d11-kube-api-access-cswmb\") pod \"maas-api-revoke-sub-850f6fb9-fq2b2\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:57.806198 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:57.806112    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:30:58.188545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:58.188513    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"]
Oct 30 22:30:58.189943 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:30:58.189913    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a31206a_fa53_4e11_86eb_f891ab1b2d11.slice/crio-678e503f7c2c2ffae3be3bb34fe1a75589f76481c217c0c1842786ec40c7f5de WatchSource:0}: Error finding container 678e503f7c2c2ffae3be3bb34fe1a75589f76481c217c0c1842786ec40c7f5de: Status 404 returned error can't find the container with id 678e503f7c2c2ffae3be3bb34fe1a75589f76481c217c0c1842786ec40c7f5de
Oct 30 22:30:58.790535 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:58.790430    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2" event={"ID":"8a31206a-fa53-4e11-86eb-f891ab1b2d11","Type":"ContainerStarted","Data":"119a727976a35ca15c126b0bff5fb803e878267a69df334ac164526c061ecce4"}
Oct 30 22:30:58.790535 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:58.790492    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2" event={"ID":"8a31206a-fa53-4e11-86eb-f891ab1b2d11","Type":"ContainerStarted","Data":"678e503f7c2c2ffae3be3bb34fe1a75589f76481c217c0c1842786ec40c7f5de"}
Oct 30 22:30:58.912851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:58.912786    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2" podStartSLOduration=1.912772131 podStartE2EDuration="1.912772131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:58.909502704 +0000 UTC m=+2470.433645471" watchObservedRunningTime="2026-10-30 22:30:58.912772131 +0000 UTC m=+2470.436914895"
Oct 30 22:30:59.799510 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:59.799473    5031 generic.go:356] "Generic (PLEG): container finished" podID="8a31206a-fa53-4e11-86eb-f891ab1b2d11" containerID="119a727976a35ca15c126b0bff5fb803e878267a69df334ac164526c061ecce4" exitCode=0
Oct 30 22:30:59.800085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:30:59.799562    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2" event={"ID":"8a31206a-fa53-4e11-86eb-f891ab1b2d11","Type":"ContainerDied","Data":"119a727976a35ca15c126b0bff5fb803e878267a69df334ac164526c061ecce4"}
Oct 30 22:31:00.967662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:00.967633    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:31:01.122077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.122042    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-openshift-ca\") pod \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") "
Oct 30 22:31:01.122313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.122108    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-xks-ca\") pod \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") "
Oct 30 22:31:01.122313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.122292    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a31206a-fa53-4e11-86eb-f891ab1b2d11-kube-api-access-cswmb\" (UniqueName: \"kubernetes.io/projected/8a31206a-fa53-4e11-86eb-f891ab1b2d11-kube-api-access-cswmb\") pod \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\" (UID: \"8a31206a-fa53-4e11-86eb-f891ab1b2d11\") "
Oct 30 22:31:01.122525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.122498    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-openshift-ca" pod "8a31206a-fa53-4e11-86eb-f891ab1b2d11" (UID: "8a31206a-fa53-4e11-86eb-f891ab1b2d11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:01.122765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.122745    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.124638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.124609    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a31206a-fa53-4e11-86eb-f891ab1b2d11-kube-api-access-cswmb" pod "8a31206a-fa53-4e11-86eb-f891ab1b2d11" (UID: "8a31206a-fa53-4e11-86eb-f891ab1b2d11"). InnerVolumeSpecName "kube-api-access-cswmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:01.124992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.124973    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-xks-ca" pod "8a31206a-fa53-4e11-86eb-f891ab1b2d11" (UID: "8a31206a-fa53-4e11-86eb-f891ab1b2d11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:01.205308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.205265    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" podUID="213406d0-a791-4e30-bd71-df34dad889ae" containerName="payload-processing" containerID="cri-o://0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7" gracePeriod=25
Oct 30 22:31:01.224654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.224605    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswmb\" (UniqueName: \"kubernetes.io/projected/8a31206a-fa53-4e11-86eb-f891ab1b2d11-kube-api-access-cswmb\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.224819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.224661    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a31206a-fa53-4e11-86eb-f891ab1b2d11-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.315190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.315120    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" podUID="12856bb0-d938-4e9e-81a4-84020b390535" containerName="payload-pre-processing" containerID="cri-o://b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4" gracePeriod=25
Oct 30 22:31:01.544866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.544773    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:31:01.819322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.819203    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"
Oct 30 22:31:01.819322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.819241    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2" event={"ID":"8a31206a-fa53-4e11-86eb-f891ab1b2d11","Type":"ContainerDied","Data":"678e503f7c2c2ffae3be3bb34fe1a75589f76481c217c0c1842786ec40c7f5de"}
Oct 30 22:31:01.819322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:01.819297    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="678e503f7c2c2ffae3be3bb34fe1a75589f76481c217c0c1842786ec40c7f5de"
Oct 30 22:31:05.272793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.272717    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main" containerID="cri-o://2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248" gracePeriod=45
Oct 30 22:31:05.551397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.551361    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:31:05.675702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.675666    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-dshm\") pod \"6ccb0670-ee31-4846-9e1c-7a1738e35208\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") "
Oct 30 22:31:05.675702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.675708    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-tmp-dir\") pod \"6ccb0670-ee31-4846-9e1c-7a1738e35208\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") "
Oct 30 22:31:05.676007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.675862    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-home\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-home\") pod \"6ccb0670-ee31-4846-9e1c-7a1738e35208\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") "
Oct 30 22:31:05.676007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.675898    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccb0670-ee31-4846-9e1c-7a1738e35208-kube-api-access-prdxt\" (UniqueName: \"kubernetes.io/projected/6ccb0670-ee31-4846-9e1c-7a1738e35208-kube-api-access-prdxt\") pod \"6ccb0670-ee31-4846-9e1c-7a1738e35208\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") "
Oct 30 22:31:05.676007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.675961    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ccb0670-ee31-4846-9e1c-7a1738e35208-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ccb0670-ee31-4846-9e1c-7a1738e35208-tls-certs\") pod \"6ccb0670-ee31-4846-9e1c-7a1738e35208\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") "
Oct 30 22:31:05.676007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.675988    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-model-cache\") pod \"6ccb0670-ee31-4846-9e1c-7a1738e35208\" (UID: \"6ccb0670-ee31-4846-9e1c-7a1738e35208\") "
Oct 30 22:31:05.676251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.676013    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-tmp-dir" pod "6ccb0670-ee31-4846-9e1c-7a1738e35208" (UID: "6ccb0670-ee31-4846-9e1c-7a1738e35208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.676251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.676136    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-home" pod "6ccb0670-ee31-4846-9e1c-7a1738e35208" (UID: "6ccb0670-ee31-4846-9e1c-7a1738e35208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.676372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.676345    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-model-cache" pod "6ccb0670-ee31-4846-9e1c-7a1738e35208" (UID: "6ccb0670-ee31-4846-9e1c-7a1738e35208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.676904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.676881    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.677013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.676907    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.677013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.676952    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.678290 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.678266    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-dshm" pod "6ccb0670-ee31-4846-9e1c-7a1738e35208" (UID: "6ccb0670-ee31-4846-9e1c-7a1738e35208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:05.678416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.678395    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccb0670-ee31-4846-9e1c-7a1738e35208-kube-api-access-prdxt" pod "6ccb0670-ee31-4846-9e1c-7a1738e35208" (UID: "6ccb0670-ee31-4846-9e1c-7a1738e35208"). InnerVolumeSpecName "kube-api-access-prdxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:05.678416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.678402    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ccb0670-ee31-4846-9e1c-7a1738e35208-tls-certs" pod "6ccb0670-ee31-4846-9e1c-7a1738e35208" (UID: "6ccb0670-ee31-4846-9e1c-7a1738e35208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:05.777799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.777758    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ccb0670-ee31-4846-9e1c-7a1738e35208-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.777799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.777794    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdxt\" (UniqueName: \"kubernetes.io/projected/6ccb0670-ee31-4846-9e1c-7a1738e35208-kube-api-access-prdxt\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.777799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.777808    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ccb0670-ee31-4846-9e1c-7a1738e35208-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.862047 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.862007    5031 generic.go:356] "Generic (PLEG): container finished" podID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerID="2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248" exitCode=0
Oct 30 22:31:05.862297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.862111    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"
Oct 30 22:31:05.862297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.862148    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" event={"ID":"6ccb0670-ee31-4846-9e1c-7a1738e35208","Type":"ContainerDied","Data":"2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248"}
Oct 30 22:31:05.862297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.862241    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq" event={"ID":"6ccb0670-ee31-4846-9e1c-7a1738e35208","Type":"ContainerDied","Data":"936b39eba72de59deda2f049c68cfa1e51e002c660239179437c1fd330ac4bcc"}
Oct 30 22:31:05.862297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.862257    5031 scope.go:122] "RemoveContainer" containerID="2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248"
Oct 30 22:31:05.876724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.876271    5031 scope.go:122] "RemoveContainer" containerID="2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248"
Oct 30 22:31:05.876724 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:31:05.876690    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248\": container with ID starting with 2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248 not found: ID does not exist" containerID="2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248"
Oct 30 22:31:05.876872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:05.876730    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248"} err="failed to get container status \"2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248\": rpc error: code = NotFound desc = could not find container \"2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248\": container with ID starting with 2b2fd6d1c23417a13092325cb433e69b50c654a2552daf4a9cefa9972990d248 not found: ID does not exist"
Oct 30 22:31:06.035189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:06.035146    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"]
Oct 30 22:31:06.044139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:06.044102    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/distinct-w3-8ca52a-kserve-5779784bb4-nx5hq"]
Oct 30 22:31:06.648691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:06.648646    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" path="/var/lib/kubelet/pods/6ccb0670-ee31-4846-9e1c-7a1738e35208/volumes"
Oct 30 22:31:09.190712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:09.190672    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" podUID="12856bb0-d938-4e9e-81a4-84020b390535" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:09.191173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:09.190729    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" podUID="213406d0-a791-4e30-bd71-df34dad889ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:12.296745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:12.296690    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:13.804577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:13.804540    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:14.086265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:14.086159    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:14.703617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:14.703585    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:19.111993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:19.111954    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"]
Oct 30 22:31:19.162655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:19.162613    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7bbc64-8t4xg"]
Oct 30 22:31:19.190486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:19.190440    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" podUID="12856bb0-d938-4e9e-81a4-84020b390535" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:19.190684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:19.190537    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" podUID="213406d0-a791-4e30-bd71-df34dad889ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:20.346610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:20.346577    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:20.543039 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:31:20.542996    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:31:20.574937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:20.574895    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:20.645225 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:20.645127    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4b21d1-8337-4416-b9a5-d05a0173053d" path="/var/lib/kubelet/pods/dc4b21d1-8337-4416-b9a5-d05a0173053d/volumes"
Oct 30 22:31:21.475438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.475049    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 22:31:21.546838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.546805    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213406d0-a791-4e30-bd71-df34dad889ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213406d0-a791-4e30-bd71-df34dad889ae-plugins-config-volume\") pod \"213406d0-a791-4e30-bd71-df34dad889ae\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") "
Oct 30 22:31:21.547056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.547001    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213406d0-a791-4e30-bd71-df34dad889ae-kube-api-access-qh65t\" (UniqueName: \"kubernetes.io/projected/213406d0-a791-4e30-bd71-df34dad889ae-kube-api-access-qh65t\") pod \"213406d0-a791-4e30-bd71-df34dad889ae\" (UID: \"213406d0-a791-4e30-bd71-df34dad889ae\") "
Oct 30 22:31:21.547326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.547300    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213406d0-a791-4e30-bd71-df34dad889ae-plugins-config-volume" pod "213406d0-a791-4e30-bd71-df34dad889ae" (UID: "213406d0-a791-4e30-bd71-df34dad889ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:21.547454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.547431    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213406d0-a791-4e30-bd71-df34dad889ae-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:21.549868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.549844    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213406d0-a791-4e30-bd71-df34dad889ae-kube-api-access-qh65t" pod "213406d0-a791-4e30-bd71-df34dad889ae" (UID: "213406d0-a791-4e30-bd71-df34dad889ae"). InnerVolumeSpecName "kube-api-access-qh65t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:21.593595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.593567    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 22:31:21.648415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.648381    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12856bb0-d938-4e9e-81a4-84020b390535-kube-api-access-pxv6g\" (UniqueName: \"kubernetes.io/projected/12856bb0-d938-4e9e-81a4-84020b390535-kube-api-access-pxv6g\") pod \"12856bb0-d938-4e9e-81a4-84020b390535\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") "
Oct 30 22:31:21.648608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.648441    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12856bb0-d938-4e9e-81a4-84020b390535-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12856bb0-d938-4e9e-81a4-84020b390535-plugins-config-volume\") pod \"12856bb0-d938-4e9e-81a4-84020b390535\" (UID: \"12856bb0-d938-4e9e-81a4-84020b390535\") "
Oct 30 22:31:21.648702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.648691    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh65t\" (UniqueName: \"kubernetes.io/projected/213406d0-a791-4e30-bd71-df34dad889ae-kube-api-access-qh65t\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:21.648822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.648801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12856bb0-d938-4e9e-81a4-84020b390535-plugins-config-volume" pod "12856bb0-d938-4e9e-81a4-84020b390535" (UID: "12856bb0-d938-4e9e-81a4-84020b390535"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:21.650702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.650684    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12856bb0-d938-4e9e-81a4-84020b390535-kube-api-access-pxv6g" pod "12856bb0-d938-4e9e-81a4-84020b390535" (UID: "12856bb0-d938-4e9e-81a4-84020b390535"). InnerVolumeSpecName "kube-api-access-pxv6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:21.749653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.749564    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12856bb0-d938-4e9e-81a4-84020b390535-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:21.749653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.749592    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxv6g\" (UniqueName: \"kubernetes.io/projected/12856bb0-d938-4e9e-81a4-84020b390535-kube-api-access-pxv6g\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:21.980878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.980835    5031 generic.go:356] "Generic (PLEG): container finished" podID="213406d0-a791-4e30-bd71-df34dad889ae" containerID="0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7" exitCode=0
Oct 30 22:31:21.980878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.980878    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"
Oct 30 22:31:21.981146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.980922    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" event={"ID":"213406d0-a791-4e30-bd71-df34dad889ae","Type":"ContainerDied","Data":"0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7"}
Oct 30 22:31:21.981146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.980980    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr" event={"ID":"213406d0-a791-4e30-bd71-df34dad889ae","Type":"ContainerDied","Data":"24471e477b900dd98ded375f95f1a67b66a48c46702701046fc124b6d59fae65"}
Oct 30 22:31:21.981146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.981007    5031 scope.go:122] "RemoveContainer" containerID="0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7"
Oct 30 22:31:21.982744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.982705    5031 generic.go:356] "Generic (PLEG): container finished" podID="12856bb0-d938-4e9e-81a4-84020b390535" containerID="b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4" exitCode=0
Oct 30 22:31:21.982900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.982753    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"
Oct 30 22:31:21.982900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.982798    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" event={"ID":"12856bb0-d938-4e9e-81a4-84020b390535","Type":"ContainerDied","Data":"b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4"}
Oct 30 22:31:21.982900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.982848    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd" event={"ID":"12856bb0-d938-4e9e-81a4-84020b390535","Type":"ContainerDied","Data":"22957ddb8e4806942e30ed7aef21ea2c203e7ed196902ed0902c18a4e5052259"}
Oct 30 22:31:21.994153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.994132    5031 scope.go:122] "RemoveContainer" containerID="0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7"
Oct 30 22:31:21.994514 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:31:21.994496    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7\": container with ID starting with 0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7 not found: ID does not exist" containerID="0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7"
Oct 30 22:31:21.994577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.994522    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7"} err="failed to get container status \"0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7\": rpc error: code = NotFound desc = could not find container \"0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7\": container with ID starting with 0f6248597f0eb16c3b30ac624e820bfd39988dfbbab64a525f7513362402e2d7 not found: ID does not exist"
Oct 30 22:31:21.994577 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:21.994538    5031 scope.go:122] "RemoveContainer" containerID="b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4"
Oct 30 22:31:22.006404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.006289    5031 scope.go:122] "RemoveContainer" containerID="b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4"
Oct 30 22:31:22.007678 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:31:22.007587    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4\": container with ID starting with b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4 not found: ID does not exist" containerID="b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4"
Oct 30 22:31:22.007678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.007635    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4"} err="failed to get container status \"b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4\": rpc error: code = NotFound desc = could not find container \"b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4\": container with ID starting with b6ca81525e5ce7418827d5deb3cf9c8e8ebc149f7e90ac8852947d334cab6cf4 not found: ID does not exist"
Oct 30 22:31:22.151782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.151746    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"]
Oct 30 22:31:22.158243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.158194    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd"]
Oct 30 22:31:22.271199 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.271111    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"]
Oct 30 22:31:22.277392 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.277360    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr"]
Oct 30 22:31:22.645953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.645904    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12856bb0-d938-4e9e-81a4-84020b390535" path="/var/lib/kubelet/pods/12856bb0-d938-4e9e-81a4-84020b390535/volumes"
Oct 30 22:31:22.646556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.646533    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213406d0-a791-4e30-bd71-df34dad889ae" path="/var/lib/kubelet/pods/213406d0-a791-4e30-bd71-df34dad889ae/volumes"
Oct 30 22:31:22.692870 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.692828    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:22.929921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.929831    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:22.980462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.980424    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"]
Oct 30 22:31:22.992656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:22.992620    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" containerID="cri-o://f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160" gracePeriod=30
Oct 30 22:31:23.161854 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:23.161825    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.468796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:24.468764    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:31:24.469204 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:24.468818    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:31:26.703372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:26.703336    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:26.942549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:26.942512    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:28.276454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.276388    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:31:28.436116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436071    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-token\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-token\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436156    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-data\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436228    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-credential-socket\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436275    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-certs\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436331    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-kube-api-access-fkj2h\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-kube-api-access-fkj2h\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436365    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-envoy\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436419    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-podinfo\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436452    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5919da72-ea5a-46a5-8fbd-32e6266099de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5919da72-ea5a-46a5-8fbd-32e6266099de-istiod-ca-cert\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436486    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-data" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.436707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436508    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-socket\") pod \"5919da72-ea5a-46a5-8fbd-32e6266099de\" (UID: \"5919da72-ea5a-46a5-8fbd-32e6266099de\") "
Oct 30 22:31:28.436707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436627    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-credential-socket" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.436707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.436686    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-socket" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.437076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.437039    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.437139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.437061    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.437266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.437161    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.437525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.437495    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-certs" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.437943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.437902    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5919da72-ea5a-46a5-8fbd-32e6266099de-istiod-ca-cert" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:28.439079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.439053    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-token" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:28.439379 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.439357    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-podinfo" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:28.439757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.439741    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-kube-api-access-fkj2h" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "kube-api-access-fkj2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:28.441232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.441176    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-envoy" pod "5919da72-ea5a-46a5-8fbd-32e6266099de" (UID: "5919da72-ea5a-46a5-8fbd-32e6266099de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:28.538380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.538281    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.538380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.538317    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkj2h\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-kube-api-access-fkj2h\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.538380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.538329    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.538380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.538338    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.538380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.538347    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5919da72-ea5a-46a5-8fbd-32e6266099de-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:28.538380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:28.538355    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5919da72-ea5a-46a5-8fbd-32e6266099de-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:29.037145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.037105    5031 generic.go:356] "Generic (PLEG): container finished" podID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerID="f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160" exitCode=0
Oct 30 22:31:29.037455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.037172    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"
Oct 30 22:31:29.037455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.037195    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" event={"ID":"5919da72-ea5a-46a5-8fbd-32e6266099de","Type":"ContainerDied","Data":"f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160"}
Oct 30 22:31:29.037455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.037296    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn" event={"ID":"5919da72-ea5a-46a5-8fbd-32e6266099de","Type":"ContainerDied","Data":"fc2df62967a12eda5e38010bfeaf9a2b276f8f43951ecb9b1a1dfe6db67df87b"}
Oct 30 22:31:29.037455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.037321    5031 scope.go:122] "RemoveContainer" containerID="f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160"
Oct 30 22:31:29.050145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.050122    5031 scope.go:122] "RemoveContainer" containerID="f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160"
Oct 30 22:31:29.050528 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:31:29.050502    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160\": container with ID starting with f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160 not found: ID does not exist" containerID="f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160"
Oct 30 22:31:29.050597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.050541    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160"} err="failed to get container status \"f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160\": rpc error: code = NotFound desc = could not find container \"f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160\": container with ID starting with f15cdf163ab99043b64043fa1fe6e43f17dc7029afb88df8bb03863eca565160 not found: ID does not exist"
Oct 30 22:31:29.260946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.260904    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"]
Oct 30 22:31:29.313230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:29.313121    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn"]
Oct 30 22:31:30.645179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:30.645138    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" path="/var/lib/kubelet/pods/5919da72-ea5a-46a5-8fbd-32e6266099de/volumes"
Oct 30 22:31:30.884467 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:30.884435    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:32.579376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:32.579333    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:31:37.710628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:37.710583    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:50.541933 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:31:50.541893    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:31:51.893363 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:51.893325    5031 scope.go:122] "RemoveContainer" containerID="c59a46dd45cc7f8236f8548902b3f0bafe2ea51c79c0f3b635b27afd971424f6"
Oct 30 22:31:51.903656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:51.903637    5031 scope.go:122] "RemoveContainer" containerID="5ed85b42ae3e4809738f376e0f0d963f6b53609246eee4f6bbe8eae1446b2bc4"
Oct 30 22:31:51.913423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:51.913404    5031 scope.go:122] "RemoveContainer" containerID="98d479e2e87be2e1230aa34eaca6da18a8725ea4f36a02ee21491ee42fa388ad"
Oct 30 22:31:51.922846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:51.922827    5031 scope.go:122] "RemoveContainer" containerID="8c2399ba11874bd56c9a50f802046f120ae89139b702128f6a8d28cff0ba43ca"
Oct 30 22:31:51.932017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:51.931988    5031 scope.go:122] "RemoveContainer" containerID="4e1fac08db124a8b7b0abab8f54d0742c68fe583b2777420534e661811507131"
Oct 30 22:31:51.941098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:51.941081    5031 scope.go:122] "RemoveContainer" containerID="612717761cc6c302b540b6633000b500b6d1fc7a7e51b748b2058edc38b3ca91"
Oct 30 22:31:53.423395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:31:53.423362    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:04.221109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:04.221074    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:06.378163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:06.378125    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:20.541820 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:32:20.541777    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:32:32.070643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:32.070596    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"]
Oct 30 22:32:32.120523 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:32.120485    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de2c5f7-2mfwj"]
Oct 30 22:32:32.646246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:32.646185    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5acb7877-6598-4b94-9298-cf965f3c2b06" path="/var/lib/kubelet/pods/5acb7877-6598-4b94-9298-cf965f3c2b06/volumes"
Oct 30 22:32:33.076377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:33.076290    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"]
Oct 30 22:32:33.126464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:33.126420    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-064e5290-22ltd"]
Oct 30 22:32:34.644877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:34.644840    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12" path="/var/lib/kubelet/pods/47d1aec2-b781-4d7f-9ad5-bdf74ebd4e12/volumes"
Oct 30 22:32:50.541761 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:32:50.541717    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:32:51.096785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:51.096751    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"]
Oct 30 22:32:51.148827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:51.148791    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"]
Oct 30 22:32:51.153251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:51.153231    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0bf654-lwmsg"]
Oct 30 22:32:51.157421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:51.157397    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab0cdc6-nxdp5"]
Oct 30 22:32:51.558952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:51.558867    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:32:52.017590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:52.017548    5031 scope.go:122] "RemoveContainer" containerID="d552f7577545c915c52062d148eaaa160c36cf48be118db0efaa88503c95a376"
Oct 30 22:32:52.031036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:52.031005    5031 scope.go:122] "RemoveContainer" containerID="1945f30cf444f0a18a934a3a462163b9b75b4fb71a118e745ad68e159c4f698a"
Oct 30 22:32:52.646319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:52.646278    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1481dbbc-6b7e-46f8-a3ab-f2e035627c07" path="/var/lib/kubelet/pods/1481dbbc-6b7e-46f8-a3ab-f2e035627c07/volumes"
Oct 30 22:32:52.646959 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:52.646935    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8" path="/var/lib/kubelet/pods/f0a4d5a6-a795-4a77-8a3a-aa6078a8bab8/volumes"
Oct 30 22:32:53.885765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:32:53.885714    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:00.069709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:00.069667    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"]
Oct 30 22:33:00.120095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:00.120058    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35c391-cdxkv"]
Oct 30 22:33:00.644778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:00.644746    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ccd031-5698-4634-a521-81f981e91e7e" path="/var/lib/kubelet/pods/31ccd031-5698-4634-a521-81f981e91e7e/volumes"
Oct 30 22:33:09.850279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:09.850236    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"]
Oct 30 22:33:09.850737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:09.850459    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m" podUID="7b9ad58e-6de2-42c1-858a-35673ce832a0" containerName="discovery" containerID="cri-o://d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04" gracePeriod=30
Oct 30 22:33:10.121251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.121222    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:33:10.169541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.169502    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-dns-cert\") pod \"7b9ad58e-6de2-42c1-858a-35673ce832a0\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") "
Oct 30 22:33:10.169751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.169601    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9ad58e-6de2-42c1-858a-35673ce832a0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9ad58e-6de2-42c1-858a-35673ce832a0-local-certs\") pod \"7b9ad58e-6de2-42c1-858a-35673ce832a0\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") "
Oct 30 22:33:10.169751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.169705    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-kubeconfig\") pod \"7b9ad58e-6de2-42c1-858a-35673ce832a0\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") "
Oct 30 22:33:10.169882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.169755    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-token\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-token\") pod \"7b9ad58e-6de2-42c1-858a-35673ce832a0\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") "
Oct 30 22:33:10.169882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.169833    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-kube-api-access-qcvvz\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-kube-api-access-qcvvz\") pod \"7b9ad58e-6de2-42c1-858a-35673ce832a0\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") "
Oct 30 22:33:10.169991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.169904    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-cacerts\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-cacerts\") pod \"7b9ad58e-6de2-42c1-858a-35673ce832a0\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") "
Oct 30 22:33:10.169991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.169948    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-ca-configmap\") pod \"7b9ad58e-6de2-42c1-858a-35673ce832a0\" (UID: \"7b9ad58e-6de2-42c1-858a-35673ce832a0\") "
Oct 30 22:33:10.170731 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.170691    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-ca-configmap" pod "7b9ad58e-6de2-42c1-858a-35673ce832a0" (UID: "7b9ad58e-6de2-42c1-858a-35673ce832a0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:10.172616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.172579    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9ad58e-6de2-42c1-858a-35673ce832a0-local-certs" pod "7b9ad58e-6de2-42c1-858a-35673ce832a0" (UID: "7b9ad58e-6de2-42c1-858a-35673ce832a0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:10.172739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.172694    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-kubeconfig" pod "7b9ad58e-6de2-42c1-858a-35673ce832a0" (UID: "7b9ad58e-6de2-42c1-858a-35673ce832a0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:10.172833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.172795    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-token" pod "7b9ad58e-6de2-42c1-858a-35673ce832a0" (UID: "7b9ad58e-6de2-42c1-858a-35673ce832a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:10.173346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.173322    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-dns-cert" pod "7b9ad58e-6de2-42c1-858a-35673ce832a0" (UID: "7b9ad58e-6de2-42c1-858a-35673ce832a0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:10.173458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.173434    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-cacerts" pod "7b9ad58e-6de2-42c1-858a-35673ce832a0" (UID: "7b9ad58e-6de2-42c1-858a-35673ce832a0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:10.174369 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.174341    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-kube-api-access-qcvvz" pod "7b9ad58e-6de2-42c1-858a-35673ce832a0" (UID: "7b9ad58e-6de2-42c1-858a-35673ce832a0"). InnerVolumeSpecName "kube-api-access-qcvvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:10.271460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.271428    5031 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-dns-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:10.271460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.271458    5031 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b9ad58e-6de2-42c1-858a-35673ce832a0-local-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:10.271460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.271467    5031 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-kubeconfig\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:10.271707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.271477    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:10.271707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.271487    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvvz\" (UniqueName: \"kubernetes.io/projected/7b9ad58e-6de2-42c1-858a-35673ce832a0-kube-api-access-qcvvz\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:10.271707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.271498    5031 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b9ad58e-6de2-42c1-858a-35673ce832a0-cacerts\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:10.271707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.271506    5031 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b9ad58e-6de2-42c1-858a-35673ce832a0-istio-csr-ca-configmap\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:10.787222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.787172    5031 generic.go:356] "Generic (PLEG): container finished" podID="7b9ad58e-6de2-42c1-858a-35673ce832a0" containerID="d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04" exitCode=0
Oct 30 22:33:10.787436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.787239    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"
Oct 30 22:33:10.787436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.787241    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m" event={"ID":"7b9ad58e-6de2-42c1-858a-35673ce832a0","Type":"ContainerDied","Data":"d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04"}
Oct 30 22:33:10.787436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.787298    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m" event={"ID":"7b9ad58e-6de2-42c1-858a-35673ce832a0","Type":"ContainerDied","Data":"14169c64d254e83941de92011d6a84e617f7279481b53e080b7775b9a04726ba"}
Oct 30 22:33:10.787436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.787322    5031 scope.go:122] "RemoveContainer" containerID="d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04"
Oct 30 22:33:10.799181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.799157    5031 scope.go:122] "RemoveContainer" containerID="d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04"
Oct 30 22:33:10.799524 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:33:10.799502    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04\": container with ID starting with d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04 not found: ID does not exist" containerID="d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04"
Oct 30 22:33:10.799576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.799535    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04"} err="failed to get container status \"d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04\": rpc error: code = NotFound desc = could not find container \"d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04\": container with ID starting with d03d3c4cab28e4d00c0941505342e50767a64bca532faf350388cedfd9a27f04 not found: ID does not exist"
Oct 30 22:33:10.962535 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:10.962438    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"]
Oct 30 22:33:11.013118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:11.013081    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m"]
Oct 30 22:33:12.645010 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:12.644973    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9ad58e-6de2-42c1-858a-35673ce832a0" path="/var/lib/kubelet/pods/7b9ad58e-6de2-42c1-858a-35673ce832a0/volumes"
Oct 30 22:33:14.443095 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:14.443037    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:33:20.591902 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:33:20.591854    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:33:21.879765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:21.879342    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:33:21.879765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:21.879421    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:24.494008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:24.493971    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:33:31.403817 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.403773    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"]
Oct 30 22:33:31.404536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404514    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:33:31.404588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404541    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec330f8c-d455-48b0-a4e6-ec008d31d305" containerName="cleanup"
Oct 30 22:33:31.404588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404563    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main"
Oct 30 22:33:31.404588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404570    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main"
Oct 30 22:33:31.404588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404584    5031 cpu_manager.go:414] "containerMap: removing container" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404591    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d2a1f3-15f5-497a-8f2a-68c9ff261f2a" containerName="cleanup"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404612    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404619    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404637    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8a31206a-fa53-4e11-86eb-f891ab1b2d11" containerName="revoke-keys"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404643    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a31206a-fa53-4e11-86eb-f891ab1b2d11" containerName="revoke-keys"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404661    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7b9ad58e-6de2-42c1-858a-35673ce832a0" containerName="discovery"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404667    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9ad58e-6de2-42c1-858a-35673ce832a0" containerName="discovery"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404689    5031 cpu_manager.go:414] "containerMap: removing container" podUID="213406d0-a791-4e30-bd71-df34dad889ae" containerName="payload-processing"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404696    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213406d0-a791-4e30-bd71-df34dad889ae" containerName="payload-processing"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404715    5031 cpu_manager.go:414] "containerMap: removing container" podUID="12856bb0-d938-4e9e-81a4-84020b390535" containerName="payload-pre-processing"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404722    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12856bb0-d938-4e9e-81a4-84020b390535" containerName="payload-pre-processing"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404734    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy"
Oct 30 22:33:31.404738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404741    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy"
Oct 30 22:33:31.405175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404857    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="213406d0-a791-4e30-bd71-df34dad889ae" containerName="payload-processing"
Oct 30 22:33:31.405175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404870    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="12856bb0-d938-4e9e-81a4-84020b390535" containerName="payload-pre-processing"
Oct 30 22:33:31.405175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404882    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5919da72-ea5a-46a5-8fbd-32e6266099de" containerName="istio-proxy"
Oct 30 22:33:31.405175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404895    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ccb0670-ee31-4846-9e1c-7a1738e35208" containerName="main"
Oct 30 22:33:31.405175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404907    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32cfce6-24a9-4ed6-bece-ead7a13b5205" containerName="cleanup"
Oct 30 22:33:31.405175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404915    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a31206a-fa53-4e11-86eb-f891ab1b2d11" containerName="revoke-keys"
Oct 30 22:33:31.405175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.404923    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9ad58e-6de2-42c1-858a-35673ce832a0" containerName="discovery"
Oct 30 22:33:31.408779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.408757    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.466068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.466028    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"]
Oct 30 22:33:31.506334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.506289    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.506546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.506496    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.506629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.506602    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6sr\" (UniqueName: \"kubernetes.io/projected/129a030b-0c4f-4cbf-9ee0-07e8583c8145-kube-api-access-ts6sr\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.513513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.513490    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:31.513513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.513493    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:31.513706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.513533    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:33:31.515240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.515202    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:31.607422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.607375    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.607637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.607478    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6sr\" (UniqueName: \"kubernetes.io/projected/129a030b-0c4f-4cbf-9ee0-07e8583c8145-kube-api-access-ts6sr\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.607637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.607559    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.608172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.608141    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.610770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.610738    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.671531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.671450    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6sr\" (UniqueName: \"kubernetes.io/projected/129a030b-0c4f-4cbf-9ee0-07e8583c8145-kube-api-access-ts6sr\") pod \"maas-api-revoke-sub-2e1f5b5f-rj4f8\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:31.725509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:31.725465    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:32.118957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:32.118930    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"]
Oct 30 22:33:32.120722 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:33:32.120697    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129a030b_0c4f_4cbf_9ee0_07e8583c8145.slice/crio-a5d95ebd5ffa695a773eacca2e0801fef709f62de4cb9606e5639d44c2e93592 WatchSource:0}: Error finding container a5d95ebd5ffa695a773eacca2e0801fef709f62de4cb9606e5639d44c2e93592: Status 404 returned error can't find the container with id a5d95ebd5ffa695a773eacca2e0801fef709f62de4cb9606e5639d44c2e93592
Oct 30 22:33:32.301099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:32.301056    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:32.981823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:32.981786    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8" event={"ID":"129a030b-0c4f-4cbf-9ee0-07e8583c8145","Type":"ContainerStarted","Data":"92f082492798f05a98092a9d8bbc1606eac229006646d98787896f00d81a9092"}
Oct 30 22:33:32.982240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:32.981852    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8" event={"ID":"129a030b-0c4f-4cbf-9ee0-07e8583c8145","Type":"ContainerStarted","Data":"a5d95ebd5ffa695a773eacca2e0801fef709f62de4cb9606e5639d44c2e93592"}
Oct 30 22:33:33.259878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:33.259782    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:33.522473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:33.522385    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:33.989595 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:33.989558    5031 generic.go:356] "Generic (PLEG): container finished" podID="129a030b-0c4f-4cbf-9ee0-07e8583c8145" containerID="92f082492798f05a98092a9d8bbc1606eac229006646d98787896f00d81a9092" exitCode=0
Oct 30 22:33:33.990067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:33.989638    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8" event={"ID":"129a030b-0c4f-4cbf-9ee0-07e8583c8145","Type":"ContainerDied","Data":"92f082492798f05a98092a9d8bbc1606eac229006646d98787896f00d81a9092"}
Oct 30 22:33:34.139115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.139077    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:34.234589 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.234549    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-openshift-ca\") pod \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") "
Oct 30 22:33:34.234802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.234597    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-xks-ca\") pod \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") "
Oct 30 22:33:34.234802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.234720    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129a030b-0c4f-4cbf-9ee0-07e8583c8145-kube-api-access-ts6sr\" (UniqueName: \"kubernetes.io/projected/129a030b-0c4f-4cbf-9ee0-07e8583c8145-kube-api-access-ts6sr\") pod \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\" (UID: \"129a030b-0c4f-4cbf-9ee0-07e8583c8145\") "
Oct 30 22:33:34.234964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.234936    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-openshift-ca" pod "129a030b-0c4f-4cbf-9ee0-07e8583c8145" (UID: "129a030b-0c4f-4cbf-9ee0-07e8583c8145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:34.235061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.235045    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:34.237063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.237032    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-xks-ca" pod "129a030b-0c4f-4cbf-9ee0-07e8583c8145" (UID: "129a030b-0c4f-4cbf-9ee0-07e8583c8145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:34.237063 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.237048    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129a030b-0c4f-4cbf-9ee0-07e8583c8145-kube-api-access-ts6sr" pod "129a030b-0c4f-4cbf-9ee0-07e8583c8145" (UID: "129a030b-0c4f-4cbf-9ee0-07e8583c8145"). InnerVolumeSpecName "kube-api-access-ts6sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:34.336500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.336462    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6sr\" (UniqueName: \"kubernetes.io/projected/129a030b-0c4f-4cbf-9ee0-07e8583c8145-kube-api-access-ts6sr\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:34.336500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.336494    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129a030b-0c4f-4cbf-9ee0-07e8583c8145-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:34.997760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.997675    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"
Oct 30 22:33:34.997760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.997703    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8" event={"ID":"129a030b-0c4f-4cbf-9ee0-07e8583c8145","Type":"ContainerDied","Data":"a5d95ebd5ffa695a773eacca2e0801fef709f62de4cb9606e5639d44c2e93592"}
Oct 30 22:33:34.997760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:34.997755    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d95ebd5ffa695a773eacca2e0801fef709f62de4cb9606e5639d44c2e93592"
Oct 30 22:33:43.928038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.927988    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"]
Oct 30 22:33:43.928766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.928742    5031 cpu_manager.go:414] "containerMap: removing container" podUID="129a030b-0c4f-4cbf-9ee0-07e8583c8145" containerName="revoke-keys"
Oct 30 22:33:43.928872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.928769    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129a030b-0c4f-4cbf-9ee0-07e8583c8145" containerName="revoke-keys"
Oct 30 22:33:43.928981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.928967    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="129a030b-0c4f-4cbf-9ee0-07e8583c8145" containerName="revoke-keys"
Oct 30 22:33:43.932341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.932317    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:43.939828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.939572    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:43.939828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.939622    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:43.939828 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.939688    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9hnk\" (UniqueName: \"kubernetes.io/projected/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-kube-api-access-k9hnk\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:43.991629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:43.991593    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"]
Oct 30 22:33:44.038587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.038546    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:44.038587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.038552    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:44.038587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.038593    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:44.038895 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.038633    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:33:44.040402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.040374    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:44.040557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.040414    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:44.040557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.040464    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9hnk\" (UniqueName: \"kubernetes.io/projected/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-kube-api-access-k9hnk\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:44.041160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.041094    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:44.043057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.043029    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:44.104732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.104699    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9hnk\" (UniqueName: \"kubernetes.io/projected/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-kube-api-access-k9hnk\") pod \"maas-api-revoke-sub-ec843018-msw42\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:44.246518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.246427    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:44.627051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:44.627020    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"]
Oct 30 22:33:44.628637 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:33:44.628608    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cf15e5_24ce_4e15_888f_8c1c19f153b3.slice/crio-91da7d1e826ef331fe2e1dea9663c3c9bc8f7c73b2cf890844bac5c83e511eab WatchSource:0}: Error finding container 91da7d1e826ef331fe2e1dea9663c3c9bc8f7c73b2cf890844bac5c83e511eab: Status 404 returned error can't find the container with id 91da7d1e826ef331fe2e1dea9663c3c9bc8f7c73b2cf890844bac5c83e511eab
Oct 30 22:33:45.073724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:45.073692    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42" event={"ID":"a8cf15e5-24ce-4e15-888f-8c1c19f153b3","Type":"ContainerStarted","Data":"2556f70fe518879e5e555f3f2127ce7cf49360d32f9ab835d82030625353cce3"}
Oct 30 22:33:45.074151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:45.073743    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42" event={"ID":"a8cf15e5-24ce-4e15-888f-8c1c19f153b3","Type":"ContainerStarted","Data":"91da7d1e826ef331fe2e1dea9663c3c9bc8f7c73b2cf890844bac5c83e511eab"}
Oct 30 22:33:45.197487 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:45.197373    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42" podStartSLOduration=2.197358332 podStartE2EDuration="2.197358332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:45.195363334 +0000 UTC m=+2636.719506123" watchObservedRunningTime="2026-10-30 22:33:45.197358332 +0000 UTC m=+2636.721501095"
Oct 30 22:33:46.081374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:46.081330    5031 generic.go:356] "Generic (PLEG): container finished" podID="a8cf15e5-24ce-4e15-888f-8c1c19f153b3" containerID="2556f70fe518879e5e555f3f2127ce7cf49360d32f9ab835d82030625353cce3" exitCode=0
Oct 30 22:33:46.081833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:46.081422    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42" event={"ID":"a8cf15e5-24ce-4e15-888f-8c1c19f153b3","Type":"ContainerDied","Data":"2556f70fe518879e5e555f3f2127ce7cf49360d32f9ab835d82030625353cce3"}
Oct 30 22:33:46.333044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:46.332942    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:33:47.263934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.263905    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:47.370321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.370283    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-xks-ca\") pod \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") "
Oct 30 22:33:47.370519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.370339    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-openshift-ca\") pod \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") "
Oct 30 22:33:47.370519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.370411    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-kube-api-access-k9hnk\" (UniqueName: \"kubernetes.io/projected/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-kube-api-access-k9hnk\") pod \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\" (UID: \"a8cf15e5-24ce-4e15-888f-8c1c19f153b3\") "
Oct 30 22:33:47.370790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.370758    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-openshift-ca" pod "a8cf15e5-24ce-4e15-888f-8c1c19f153b3" (UID: "a8cf15e5-24ce-4e15-888f-8c1c19f153b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:47.370879 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.370835    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:47.372690 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.372659    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-kube-api-access-k9hnk" pod "a8cf15e5-24ce-4e15-888f-8c1c19f153b3" (UID: "a8cf15e5-24ce-4e15-888f-8c1c19f153b3"). InnerVolumeSpecName "kube-api-access-k9hnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:47.372813 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.372748    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-xks-ca" pod "a8cf15e5-24ce-4e15-888f-8c1c19f153b3" (UID: "a8cf15e5-24ce-4e15-888f-8c1c19f153b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:47.472222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.472180    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9hnk\" (UniqueName: \"kubernetes.io/projected/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-kube-api-access-k9hnk\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:47.472222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:47.472227    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8cf15e5-24ce-4e15-888f-8c1c19f153b3-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:48.096986 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:48.096954    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"
Oct 30 22:33:48.097230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:48.096980    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42" event={"ID":"a8cf15e5-24ce-4e15-888f-8c1c19f153b3","Type":"ContainerDied","Data":"91da7d1e826ef331fe2e1dea9663c3c9bc8f7c73b2cf890844bac5c83e511eab"}
Oct 30 22:33:48.097230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:48.097035    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91da7d1e826ef331fe2e1dea9663c3c9bc8f7c73b2cf890844bac5c83e511eab"
Oct 30 22:33:50.546460 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:33:50.546414    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:33:52.077734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:52.077692    5031 scope.go:122] "RemoveContainer" containerID="fb00fbfabdff377197c2109730b917c04887f2212c62b8d6fc38ac4ee6090662"
Oct 30 22:33:52.089948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:52.089923    5031 scope.go:122] "RemoveContainer" containerID="bcfc36b25e84bb7ffa7baa8ad53286c39eda4384b201e85c6e1834170da140a7"
Oct 30 22:33:52.100132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:52.100108    5031 scope.go:122] "RemoveContainer" containerID="6eebf0aa062c762cbe3b7953b22bfd25e983612441272e72232eb9a4be1795d8"
Oct 30 22:33:52.111167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:52.111135    5031 scope.go:122] "RemoveContainer" containerID="d632da1e70f7e2a01bcf93800437490c70fbc12a3c9c7e8816e57c4ee7ab4572"
Oct 30 22:33:52.123415 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:52.123387    5031 scope.go:122] "RemoveContainer" containerID="bb282985019d0b0b6bb07b7698b3439d50494539c7e12da8e7c44cecefacebe7"
Oct 30 22:33:52.133865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:52.133844    5031 scope.go:122] "RemoveContainer" containerID="b26b64095fdd46dd9f6820e8599c3f7d4a43da3b95fd09d092d48601eca95cf2"
Oct 30 22:33:52.143991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:52.143967    5031 scope.go:122] "RemoveContainer" containerID="e7d3ebc3b564965f527d37b6256aa29b9c3e1d6849e5760581778b837fd612ce"
Oct 30 22:33:54.199166 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:54.199132    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:54.437582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:54.437547    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:54.677372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:54.677342    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:58.063127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:58.063021    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"]
Oct 30 22:33:58.115468 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:58.115428    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"]
Oct 30 22:33:58.118611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:58.118580    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a85087d-8z8nb"]
Oct 30 22:33:58.121032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:58.121007    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75526bb9-f92pd"]
Oct 30 22:33:58.645450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:58.645413    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a25f616-b824-4b62-ab6e-3ae17cc6f568" path="/var/lib/kubelet/pods/2a25f616-b824-4b62-ab6e-3ae17cc6f568/volumes"
Oct 30 22:33:58.645861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:33:58.645848    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c0d458-d1b5-44a5-928e-3f678f5fcf61" path="/var/lib/kubelet/pods/36c0d458-d1b5-44a5-928e-3f678f5fcf61/volumes"
Oct 30 22:34:01.763340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:01.763290    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:34:20.541952 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:34:20.541912    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:34:22.550404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.550366    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"]
Oct 30 22:34:22.550886 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.550872    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a8cf15e5-24ce-4e15-888f-8c1c19f153b3" containerName="revoke-keys"
Oct 30 22:34:22.550942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.550887    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cf15e5-24ce-4e15-888f-8c1c19f153b3" containerName="revoke-keys"
Oct 30 22:34:22.550991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.550982    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cf15e5-24ce-4e15-888f-8c1c19f153b3" containerName="revoke-keys"
Oct 30 22:34:22.554589 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.554572    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.613951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.613912    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"]
Oct 30 22:34:22.661260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.661197    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:34:22.661260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.661197    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:22.661260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.661253    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:22.661520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.661308    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:22.682427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.682391    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.682620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.682442    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.682672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.682637    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmw7l\" (UniqueName: \"kubernetes.io/projected/406836bb-78b6-4fd4-ba41-703aff195b31-kube-api-access-jmw7l\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.783449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.783412    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.783659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.783459    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.783659 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.783537    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmw7l\" (UniqueName: \"kubernetes.io/projected/406836bb-78b6-4fd4-ba41-703aff195b31-kube-api-access-jmw7l\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.784108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.784074    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.786243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.786201    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.846952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.846922    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmw7l\" (UniqueName: \"kubernetes.io/projected/406836bb-78b6-4fd4-ba41-703aff195b31-kube-api-access-jmw7l\") pod \"maas-api-revoke-sub-046e1343-hlwrf\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:22.868049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:22.868020    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:23.249132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:23.249101    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"]
Oct 30 22:34:23.251020 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:34:23.250974    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406836bb_78b6_4fd4_ba41_703aff195b31.slice/crio-245c23582eee16633b8d7605921b2bdc81dab93badec8d4713264ee657f7bc74 WatchSource:0}: Error finding container 245c23582eee16633b8d7605921b2bdc81dab93badec8d4713264ee657f7bc74: Status 404 returned error can't find the container with id 245c23582eee16633b8d7605921b2bdc81dab93badec8d4713264ee657f7bc74
Oct 30 22:34:23.367514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:23.367482    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf" event={"ID":"406836bb-78b6-4fd4-ba41-703aff195b31","Type":"ContainerStarted","Data":"245c23582eee16633b8d7605921b2bdc81dab93badec8d4713264ee657f7bc74"}
Oct 30 22:34:24.375494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:24.375402    5031 generic.go:356] "Generic (PLEG): container finished" podID="406836bb-78b6-4fd4-ba41-703aff195b31" containerID="a563873edbfc151318ea595d209a30eda5333b2302d722ae483504cb186efb30" exitCode=0
Oct 30 22:34:24.375494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:24.375467    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf" event={"ID":"406836bb-78b6-4fd4-ba41-703aff195b31","Type":"ContainerDied","Data":"a563873edbfc151318ea595d209a30eda5333b2302d722ae483504cb186efb30"}
Oct 30 22:34:25.530032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.530005    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:25.613851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.613817    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-xks-ca\") pod \"406836bb-78b6-4fd4-ba41-703aff195b31\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") "
Oct 30 22:34:25.613851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.613859    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406836bb-78b6-4fd4-ba41-703aff195b31-kube-api-access-jmw7l\" (UniqueName: \"kubernetes.io/projected/406836bb-78b6-4fd4-ba41-703aff195b31-kube-api-access-jmw7l\") pod \"406836bb-78b6-4fd4-ba41-703aff195b31\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") "
Oct 30 22:34:25.614150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.613886    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-openshift-ca\") pod \"406836bb-78b6-4fd4-ba41-703aff195b31\" (UID: \"406836bb-78b6-4fd4-ba41-703aff195b31\") "
Oct 30 22:34:25.614327 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.614302    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-openshift-ca" pod "406836bb-78b6-4fd4-ba41-703aff195b31" (UID: "406836bb-78b6-4fd4-ba41-703aff195b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:25.616299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.616263    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-xks-ca" pod "406836bb-78b6-4fd4-ba41-703aff195b31" (UID: "406836bb-78b6-4fd4-ba41-703aff195b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:25.616410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.616329    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406836bb-78b6-4fd4-ba41-703aff195b31-kube-api-access-jmw7l" pod "406836bb-78b6-4fd4-ba41-703aff195b31" (UID: "406836bb-78b6-4fd4-ba41-703aff195b31"). InnerVolumeSpecName "kube-api-access-jmw7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:25.715747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.715641    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:34:25.715747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.715684    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmw7l\" (UniqueName: \"kubernetes.io/projected/406836bb-78b6-4fd4-ba41-703aff195b31-kube-api-access-jmw7l\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:34:25.715747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:25.715703    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406836bb-78b6-4fd4-ba41-703aff195b31-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:34:26.391686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:26.391653    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"
Oct 30 22:34:26.391686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:26.391668    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf" event={"ID":"406836bb-78b6-4fd4-ba41-703aff195b31","Type":"ContainerDied","Data":"245c23582eee16633b8d7605921b2bdc81dab93badec8d4713264ee657f7bc74"}
Oct 30 22:34:26.391993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:26.391730    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245c23582eee16633b8d7605921b2bdc81dab93badec8d4713264ee657f7bc74"
Oct 30 22:34:33.328491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.328452    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"]
Oct 30 22:34:33.328994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.328981    5031 cpu_manager.go:414] "containerMap: removing container" podUID="406836bb-78b6-4fd4-ba41-703aff195b31" containerName="revoke-keys"
Oct 30 22:34:33.329041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.328996    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406836bb-78b6-4fd4-ba41-703aff195b31" containerName="revoke-keys"
Oct 30 22:34:33.329090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.329081    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="406836bb-78b6-4fd4-ba41-703aff195b31" containerName="revoke-keys"
Oct 30 22:34:33.332498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.332483    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.392180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.392145    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"]
Oct 30 22:34:33.438759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.438721    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:33.438937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.438837    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:34:33.438937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.438874    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:33.438937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.438882    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:33.509372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.509333    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.509563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.509410    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ctc4\" (UniqueName: \"kubernetes.io/projected/1f52ab89-63a0-4741-bdad-689c0bce186f-kube-api-access-8ctc4\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.509563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.509453    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.611031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.610994    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.611246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.611062    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ctc4\" (UniqueName: \"kubernetes.io/projected/1f52ab89-63a0-4741-bdad-689c0bce186f-kube-api-access-8ctc4\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.611246 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.611088    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.611880 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.611859    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.613676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.613649    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.675295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.675267    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ctc4\" (UniqueName: \"kubernetes.io/projected/1f52ab89-63a0-4741-bdad-689c0bce186f-kube-api-access-8ctc4\") pod \"maas-api-revoke-sub-d3fba056-8xgfk\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:33.946534 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:33.946452    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:34.329407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:34.329381    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"]
Oct 30 22:34:34.331291 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:34:34.331266    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f52ab89_63a0_4741_bdad_689c0bce186f.slice/crio-7f32b62c811d31d0d5315b26dfe5e4f4c742eac588b0b0d9de1de8476b8c78e6 WatchSource:0}: Error finding container 7f32b62c811d31d0d5315b26dfe5e4f4c742eac588b0b0d9de1de8476b8c78e6: Status 404 returned error can't find the container with id 7f32b62c811d31d0d5315b26dfe5e4f4c742eac588b0b0d9de1de8476b8c78e6
Oct 30 22:34:34.443822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:34.443782    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk" event={"ID":"1f52ab89-63a0-4741-bdad-689c0bce186f","Type":"ContainerStarted","Data":"7f32b62c811d31d0d5315b26dfe5e4f4c742eac588b0b0d9de1de8476b8c78e6"}
Oct 30 22:34:35.452175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:35.452140    5031 generic.go:356] "Generic (PLEG): container finished" podID="1f52ab89-63a0-4741-bdad-689c0bce186f" containerID="534d083fc4156249ce96cd1b2b9eb1094fdb78fbf41e320585d34dfea59650e2" exitCode=0
Oct 30 22:34:35.452686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:35.452173    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk" event={"ID":"1f52ab89-63a0-4741-bdad-689c0bce186f","Type":"ContainerDied","Data":"534d083fc4156249ce96cd1b2b9eb1094fdb78fbf41e320585d34dfea59650e2"}
Oct 30 22:34:35.465484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:35.465460    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:34:36.633928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.633890    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:36.787288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.785152    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f52ab89-63a0-4741-bdad-689c0bce186f-kube-api-access-8ctc4\" (UniqueName: \"kubernetes.io/projected/1f52ab89-63a0-4741-bdad-689c0bce186f-kube-api-access-8ctc4\") pod \"1f52ab89-63a0-4741-bdad-689c0bce186f\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") "
Oct 30 22:34:36.787288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.785267    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-openshift-ca\") pod \"1f52ab89-63a0-4741-bdad-689c0bce186f\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") "
Oct 30 22:34:36.787288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.785329    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-xks-ca\") pod \"1f52ab89-63a0-4741-bdad-689c0bce186f\" (UID: \"1f52ab89-63a0-4741-bdad-689c0bce186f\") "
Oct 30 22:34:36.793299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.792408    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-openshift-ca" pod "1f52ab89-63a0-4741-bdad-689c0bce186f" (UID: "1f52ab89-63a0-4741-bdad-689c0bce186f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:36.794537 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.794509    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f52ab89-63a0-4741-bdad-689c0bce186f-kube-api-access-8ctc4" pod "1f52ab89-63a0-4741-bdad-689c0bce186f" (UID: "1f52ab89-63a0-4741-bdad-689c0bce186f"). InnerVolumeSpecName "kube-api-access-8ctc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:36.796536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.796505    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-xks-ca" pod "1f52ab89-63a0-4741-bdad-689c0bce186f" (UID: "1f52ab89-63a0-4741-bdad-689c0bce186f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:36.886412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.886364    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ctc4\" (UniqueName: \"kubernetes.io/projected/1f52ab89-63a0-4741-bdad-689c0bce186f-kube-api-access-8ctc4\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.886412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.886409    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.886649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:36.886426    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f52ab89-63a0-4741-bdad-689c0bce186f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:34:37.514263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:37.510796    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk" event={"ID":"1f52ab89-63a0-4741-bdad-689c0bce186f","Type":"ContainerDied","Data":"7f32b62c811d31d0d5315b26dfe5e4f4c742eac588b0b0d9de1de8476b8c78e6"}
Oct 30 22:34:37.514263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:37.510868    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f32b62c811d31d0d5315b26dfe5e4f4c742eac588b0b0d9de1de8476b8c78e6"
Oct 30 22:34:37.514263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:37.510947    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"
Oct 30 22:34:48.075466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:48.075421    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:34:49.953237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:49.952723    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:34:49.988236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:49.988041    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:34:50.542106 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:34:50.542070    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:34:52.205137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:52.205095    5031 scope.go:122] "RemoveContainer" containerID="d59cfe05f3c4da12724fa9281a338d15434905441b2f8966fc570ee42463f28e"
Oct 30 22:34:52.215219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:52.215184    5031 scope.go:122] "RemoveContainer" containerID="fa8d9e7dea2e48832c3eb6532e88f2824f1195d11e1e43a7e27bf5baef83cd59"
Oct 30 22:34:54.604865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:54.604815    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:56.021239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:56.018961    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nqjll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:56.021239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:56.019052    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll" podUID="b622abc4-5b50-4f1b-84cc-30edabc33258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:58.360527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:34:58.360483    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:35:11.388182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.388139    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"]
Oct 30 22:35:11.388892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.388868    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1f52ab89-63a0-4741-bdad-689c0bce186f" containerName="revoke-keys"
Oct 30 22:35:11.388998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.388894    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f52ab89-63a0-4741-bdad-689c0bce186f" containerName="revoke-keys"
Oct 30 22:35:11.389088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.389074    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f52ab89-63a0-4741-bdad-689c0bce186f" containerName="revoke-keys"
Oct 30 22:35:11.393921 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.393896    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.455227 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.455170    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"]
Oct 30 22:35:11.484309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.484267    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.484498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.484413    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.484498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.484489    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knl2\" (UniqueName: \"kubernetes.io/projected/e149abe5-35e5-4a37-b477-55928db55f90-kube-api-access-6knl2\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.501802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.501771    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:11.501996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.501978    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:35:11.502048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.502011    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:35:11.502456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.502441    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:35:11.586042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.586004    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.586284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.586078    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.586284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.586136    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knl2\" (UniqueName: \"kubernetes.io/projected/e149abe5-35e5-4a37-b477-55928db55f90-kube-api-access-6knl2\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.586744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.586720    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.588779 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.588755    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.650734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.650661    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knl2\" (UniqueName: \"kubernetes.io/projected/e149abe5-35e5-4a37-b477-55928db55f90-kube-api-access-6knl2\") pod \"maas-api-revoke-sub-ae594979-8wwkq\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:11.708242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:11.708197    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:12.090588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:12.090562    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"]
Oct 30 22:35:12.091767 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:35:12.091739    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode149abe5_35e5_4a37_b477_55928db55f90.slice/crio-99c86c81a3e75eae71bcbe1624dca4f837bc4a4def1c0ef0e9c63d408e9c051d WatchSource:0}: Error finding container 99c86c81a3e75eae71bcbe1624dca4f837bc4a4def1c0ef0e9c63d408e9c051d: Status 404 returned error can't find the container with id 99c86c81a3e75eae71bcbe1624dca4f837bc4a4def1c0ef0e9c63d408e9c051d
Oct 30 22:35:12.804447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:12.804345    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq" event={"ID":"e149abe5-35e5-4a37-b477-55928db55f90","Type":"ContainerStarted","Data":"44359118473f79a0fb2dde2199f7007aebfd955d51bb7ad05c4ccf89d81ce67d"}
Oct 30 22:35:12.804447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:12.804404    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq" event={"ID":"e149abe5-35e5-4a37-b477-55928db55f90","Type":"ContainerStarted","Data":"99c86c81a3e75eae71bcbe1624dca4f837bc4a4def1c0ef0e9c63d408e9c051d"}
Oct 30 22:35:12.926243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:12.926175    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq" podStartSLOduration=1.9261612879999999 podStartE2EDuration="1.926161288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:12.923521353 +0000 UTC m=+2724.447664119" watchObservedRunningTime="2026-10-30 22:35:12.926161288 +0000 UTC m=+2724.450304051"
Oct 30 22:35:13.812515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:13.812477    5031 generic.go:356] "Generic (PLEG): container finished" podID="e149abe5-35e5-4a37-b477-55928db55f90" containerID="44359118473f79a0fb2dde2199f7007aebfd955d51bb7ad05c4ccf89d81ce67d" exitCode=0
Oct 30 22:35:13.813058 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:13.812562    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq" event={"ID":"e149abe5-35e5-4a37-b477-55928db55f90","Type":"ContainerDied","Data":"44359118473f79a0fb2dde2199f7007aebfd955d51bb7ad05c4ccf89d81ce67d"}
Oct 30 22:35:14.969806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:14.969781    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:15.078771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.078680    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"]
Oct 30 22:35:15.081220 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.081185    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a03aa0f-krnx5"]
Oct 30 22:35:15.124780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.124751    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-openshift-ca\") pod \"e149abe5-35e5-4a37-b477-55928db55f90\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") "
Oct 30 22:35:15.124988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.124798    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-xks-ca\") pod \"e149abe5-35e5-4a37-b477-55928db55f90\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") "
Oct 30 22:35:15.124988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.124891    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e149abe5-35e5-4a37-b477-55928db55f90-kube-api-access-6knl2\" (UniqueName: \"kubernetes.io/projected/e149abe5-35e5-4a37-b477-55928db55f90-kube-api-access-6knl2\") pod \"e149abe5-35e5-4a37-b477-55928db55f90\" (UID: \"e149abe5-35e5-4a37-b477-55928db55f90\") "
Oct 30 22:35:15.125272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.125240    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-openshift-ca" pod "e149abe5-35e5-4a37-b477-55928db55f90" (UID: "e149abe5-35e5-4a37-b477-55928db55f90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:15.127539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.127508    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-xks-ca" pod "e149abe5-35e5-4a37-b477-55928db55f90" (UID: "e149abe5-35e5-4a37-b477-55928db55f90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:15.129319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.129292    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e149abe5-35e5-4a37-b477-55928db55f90-kube-api-access-6knl2" pod "e149abe5-35e5-4a37-b477-55928db55f90" (UID: "e149abe5-35e5-4a37-b477-55928db55f90"). InnerVolumeSpecName "kube-api-access-6knl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:15.226689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.226656    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.226689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.226687    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e149abe5-35e5-4a37-b477-55928db55f90-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.226689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.226697    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knl2\" (UniqueName: \"kubernetes.io/projected/e149abe5-35e5-4a37-b477-55928db55f90-kube-api-access-6knl2\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.826984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.826952    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"
Oct 30 22:35:15.827162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.826979    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq" event={"ID":"e149abe5-35e5-4a37-b477-55928db55f90","Type":"ContainerDied","Data":"99c86c81a3e75eae71bcbe1624dca4f837bc4a4def1c0ef0e9c63d408e9c051d"}
Oct 30 22:35:15.827162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:15.827030    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99c86c81a3e75eae71bcbe1624dca4f837bc4a4def1c0ef0e9c63d408e9c051d"
Oct 30 22:35:16.644703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:16.644671    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da3cdce-0288-42ed-9579-e8c239783ba3" path="/var/lib/kubelet/pods/0da3cdce-0288-42ed-9579-e8c239783ba3/volumes"
Oct 30 22:35:20.542289 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:35:20.542246    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:35:24.334937 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:24.334898    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:35:34.265380 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:34.265339    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:35:39.629179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.629139    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"]
Oct 30 22:35:39.629716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.629678    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e149abe5-35e5-4a37-b477-55928db55f90" containerName="revoke-keys"
Oct 30 22:35:39.629716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.629692    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e149abe5-35e5-4a37-b477-55928db55f90" containerName="revoke-keys"
Oct 30 22:35:39.629873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.629776    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e149abe5-35e5-4a37-b477-55928db55f90" containerName="revoke-keys"
Oct 30 22:35:39.634200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.634175    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.691776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.691741    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"]
Oct 30 22:35:39.701038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.701003    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.701264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.701061    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.701264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.701243    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mz57\" (UniqueName: \"kubernetes.io/projected/d61423af-9f80-4851-a97d-d695d87ac0cb-kube-api-access-8mz57\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.738753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.738706    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:39.738949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.738765    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:35:39.738949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.738763    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:35:39.738949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.738944    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:35:39.802751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.802720    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.802751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.802762    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.802994 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.802829    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mz57\" (UniqueName: \"kubernetes.io/projected/d61423af-9f80-4851-a97d-d695d87ac0cb-kube-api-access-8mz57\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.803551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.803531    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.805397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.805372    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.866770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.866741    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mz57\" (UniqueName: \"kubernetes.io/projected/d61423af-9f80-4851-a97d-d695d87ac0cb-kube-api-access-8mz57\") pod \"maas-api-revoke-sub-a4aef7d7-pqnzv\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:39.958245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:39.958132    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:40.343853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:40.343804    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"]
Oct 30 22:35:40.345084 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:35:40.345042    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61423af_9f80_4851_a97d_d695d87ac0cb.slice/crio-b9572de45a893904b7d7ea42e2211fd5a7e131fac8fc5156ab848c924c40a1ee WatchSource:0}: Error finding container b9572de45a893904b7d7ea42e2211fd5a7e131fac8fc5156ab848c924c40a1ee: Status 404 returned error can't find the container with id b9572de45a893904b7d7ea42e2211fd5a7e131fac8fc5156ab848c924c40a1ee
Oct 30 22:35:41.049313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:41.049218    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv" event={"ID":"d61423af-9f80-4851-a97d-d695d87ac0cb","Type":"ContainerStarted","Data":"546ac43654bd78ba636f011bd71002c60ebc5da070275456335d30c34ce078ec"}
Oct 30 22:35:41.049313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:41.049275    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv" event={"ID":"d61423af-9f80-4851-a97d-d695d87ac0cb","Type":"ContainerStarted","Data":"b9572de45a893904b7d7ea42e2211fd5a7e131fac8fc5156ab848c924c40a1ee"}
Oct 30 22:35:41.185719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:41.185658    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv" podStartSLOduration=2.185643818 podStartE2EDuration="2.185643818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:41.185361939 +0000 UTC m=+2752.709504722" watchObservedRunningTime="2026-10-30 22:35:41.185643818 +0000 UTC m=+2752.709786582"
Oct 30 22:35:42.057085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:42.057047    5031 generic.go:356] "Generic (PLEG): container finished" podID="d61423af-9f80-4851-a97d-d695d87ac0cb" containerID="546ac43654bd78ba636f011bd71002c60ebc5da070275456335d30c34ce078ec" exitCode=0
Oct 30 22:35:42.057573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:42.057139    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv" event={"ID":"d61423af-9f80-4851-a97d-d695d87ac0cb","Type":"ContainerDied","Data":"546ac43654bd78ba636f011bd71002c60ebc5da070275456335d30c34ce078ec"}
Oct 30 22:35:43.217278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.217252    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:43.341638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.341556    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61423af-9f80-4851-a97d-d695d87ac0cb-kube-api-access-8mz57\" (UniqueName: \"kubernetes.io/projected/d61423af-9f80-4851-a97d-d695d87ac0cb-kube-api-access-8mz57\") pod \"d61423af-9f80-4851-a97d-d695d87ac0cb\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") "
Oct 30 22:35:43.341834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.341712    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-openshift-ca\") pod \"d61423af-9f80-4851-a97d-d695d87ac0cb\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") "
Oct 30 22:35:43.341834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.341771    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-xks-ca\") pod \"d61423af-9f80-4851-a97d-d695d87ac0cb\" (UID: \"d61423af-9f80-4851-a97d-d695d87ac0cb\") "
Oct 30 22:35:43.342127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.342098    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-openshift-ca" pod "d61423af-9f80-4851-a97d-d695d87ac0cb" (UID: "d61423af-9f80-4851-a97d-d695d87ac0cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:43.344037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.343999    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61423af-9f80-4851-a97d-d695d87ac0cb-kube-api-access-8mz57" pod "d61423af-9f80-4851-a97d-d695d87ac0cb" (UID: "d61423af-9f80-4851-a97d-d695d87ac0cb"). InnerVolumeSpecName "kube-api-access-8mz57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:43.344186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.344111    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-xks-ca" pod "d61423af-9f80-4851-a97d-d695d87ac0cb" (UID: "d61423af-9f80-4851-a97d-d695d87ac0cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:43.442888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.442848    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mz57\" (UniqueName: \"kubernetes.io/projected/d61423af-9f80-4851-a97d-d695d87ac0cb-kube-api-access-8mz57\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:43.442888 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.442885    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:43.443123 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:43.442900    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61423af-9f80-4851-a97d-d695d87ac0cb-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:35:44.071452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:44.071412    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv" event={"ID":"d61423af-9f80-4851-a97d-d695d87ac0cb","Type":"ContainerDied","Data":"b9572de45a893904b7d7ea42e2211fd5a7e131fac8fc5156ab848c924c40a1ee"}
Oct 30 22:35:44.071452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:44.071436    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"
Oct 30 22:35:44.071738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:44.071464    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9572de45a893904b7d7ea42e2211fd5a7e131fac8fc5156ab848c924c40a1ee"
Oct 30 22:35:50.542694 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:35:50.542651    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:35:52.244142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:52.244104    5031 scope.go:122] "RemoveContainer" containerID="6c268284d7ee8686ad96ce46721d1c7cf6afd173cd0e9de22c01438e9919521d"
Oct 30 22:35:52.254235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:52.254195    5031 scope.go:122] "RemoveContainer" containerID="07c42e141315f7557197b6685b922638e885434d782f91d74d0b374588157b9a"
Oct 30 22:35:52.266232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:52.266190    5031 scope.go:122] "RemoveContainer" containerID="64f628f85166dc06b70e3b2f86e07dceed98c227faf1793b2c2c73a252f92ab2"
Oct 30 22:35:52.276633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:52.276610    5031 scope.go:122] "RemoveContainer" containerID="0e49c046ddb8bbe4d10df2c16f52b99725715db86752d7a9af7389f6146d0938"
Oct 30 22:35:52.563045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:52.562950    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:35:54.897076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:54.897042    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:58.193768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:35:58.193735    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.075762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:01.075711    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"]
Oct 30 22:36:01.136235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:01.134918    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850f6fb9-fq2b2"]
Oct 30 22:36:02.645962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:02.645920    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a31206a-fa53-4e11-86eb-f891ab1b2d11" path="/var/lib/kubelet/pods/8a31206a-fa53-4e11-86eb-f891ab1b2d11/volumes"
Oct 30 22:36:05.595276 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:05.595238    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:36:13.475754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.475716    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"]
Oct 30 22:36:13.476280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.476265    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d61423af-9f80-4851-a97d-d695d87ac0cb" containerName="revoke-keys"
Oct 30 22:36:13.476280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.476282    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61423af-9f80-4851-a97d-d695d87ac0cb" containerName="revoke-keys"
Oct 30 22:36:13.476404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.476394    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61423af-9f80-4851-a97d-d695d87ac0cb" containerName="revoke-keys"
Oct 30 22:36:13.481306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.481286    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.586113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.586068    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:36:13.586347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.586141    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:36:13.586347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.586197    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:36:13.589089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.589064    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"]
Oct 30 22:36:13.622782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.622744    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rf2p\" (UniqueName: \"kubernetes.io/projected/b05b4c27-4862-41d2-96ce-35b1a941bb3e-kube-api-access-5rf2p\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.622980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.622805    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.622980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.622920    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.636910 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.636875    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:36:13.724282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.724203    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.724488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.724417    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rf2p\" (UniqueName: \"kubernetes.io/projected/b05b4c27-4862-41d2-96ce-35b1a941bb3e-kube-api-access-5rf2p\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.724488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.724472    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.725133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.725104    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.726890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.726843    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.788322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.788289    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rf2p\" (UniqueName: \"kubernetes.io/projected/b05b4c27-4862-41d2-96ce-35b1a941bb3e-kube-api-access-5rf2p\") pod \"maas-api-revoke-sub-79805ca9-lvj92\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:13.795091 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:13.795065    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:14.182165 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:14.182134    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"]
Oct 30 22:36:14.184275 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:36:14.184242    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05b4c27_4862_41d2_96ce_35b1a941bb3e.slice/crio-a69a1e585deb8901ddc0ef4a7bb84ea67bb7c00b9dd1e7ebb39c176325af97cd WatchSource:0}: Error finding container a69a1e585deb8901ddc0ef4a7bb84ea67bb7c00b9dd1e7ebb39c176325af97cd: Status 404 returned error can't find the container with id a69a1e585deb8901ddc0ef4a7bb84ea67bb7c00b9dd1e7ebb39c176325af97cd
Oct 30 22:36:14.279195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:14.279156    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92" event={"ID":"b05b4c27-4862-41d2-96ce-35b1a941bb3e","Type":"ContainerStarted","Data":"a69a1e585deb8901ddc0ef4a7bb84ea67bb7c00b9dd1e7ebb39c176325af97cd"}
Oct 30 22:36:15.287701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:15.287666    5031 generic.go:356] "Generic (PLEG): container finished" podID="b05b4c27-4862-41d2-96ce-35b1a941bb3e" containerID="a2f5575caafa1b186183c8afabd9f3f0114fc6c791d312fa8271cdb5a229ed14" exitCode=0
Oct 30 22:36:15.288132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:15.287752    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92" event={"ID":"b05b4c27-4862-41d2-96ce-35b1a941bb3e","Type":"ContainerDied","Data":"a2f5575caafa1b186183c8afabd9f3f0114fc6c791d312fa8271cdb5a229ed14"}
Oct 30 22:36:16.452447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.452419    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:16.554538 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.554443    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-xks-ca\") pod \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") "
Oct 30 22:36:16.554706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.554585    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05b4c27-4862-41d2-96ce-35b1a941bb3e-kube-api-access-5rf2p\" (UniqueName: \"kubernetes.io/projected/b05b4c27-4862-41d2-96ce-35b1a941bb3e-kube-api-access-5rf2p\") pod \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") "
Oct 30 22:36:16.554706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.554622    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-openshift-ca\") pod \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\" (UID: \"b05b4c27-4862-41d2-96ce-35b1a941bb3e\") "
Oct 30 22:36:16.555050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.555027    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-openshift-ca" pod "b05b4c27-4862-41d2-96ce-35b1a941bb3e" (UID: "b05b4c27-4862-41d2-96ce-35b1a941bb3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:16.556837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.556803    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05b4c27-4862-41d2-96ce-35b1a941bb3e-kube-api-access-5rf2p" pod "b05b4c27-4862-41d2-96ce-35b1a941bb3e" (UID: "b05b4c27-4862-41d2-96ce-35b1a941bb3e"). InnerVolumeSpecName "kube-api-access-5rf2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:16.556962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.556862    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-xks-ca" pod "b05b4c27-4862-41d2-96ce-35b1a941bb3e" (UID: "b05b4c27-4862-41d2-96ce-35b1a941bb3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:16.655862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.655830    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rf2p\" (UniqueName: \"kubernetes.io/projected/b05b4c27-4862-41d2-96ce-35b1a941bb3e-kube-api-access-5rf2p\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.655977 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.655865    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.655977 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:16.655875    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05b4c27-4862-41d2-96ce-35b1a941bb3e-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:17.301447 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:17.301415    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"
Oct 30 22:36:17.301636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:17.301423    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92" event={"ID":"b05b4c27-4862-41d2-96ce-35b1a941bb3e","Type":"ContainerDied","Data":"a69a1e585deb8901ddc0ef4a7bb84ea67bb7c00b9dd1e7ebb39c176325af97cd"}
Oct 30 22:36:17.301636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:17.301538    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a69a1e585deb8901ddc0ef4a7bb84ea67bb7c00b9dd1e7ebb39c176325af97cd"
Oct 30 22:36:20.541994 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:36:20.541943    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:36:28.161225 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:28.161180    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:36:37.384868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.384825    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"]
Oct 30 22:36:37.385509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.385329    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b05b4c27-4862-41d2-96ce-35b1a941bb3e" containerName="revoke-keys"
Oct 30 22:36:37.385509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.385341    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05b4c27-4862-41d2-96ce-35b1a941bb3e" containerName="revoke-keys"
Oct 30 22:36:37.385509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.385437    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05b4c27-4862-41d2-96ce-35b1a941bb3e" containerName="revoke-keys"
Oct 30 22:36:37.388891 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.388874    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.448076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.448046    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"]
Oct 30 22:36:37.495187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.495153    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:36:37.495385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.495286    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:36:37.495385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.495302    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:36:37.495385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.495320    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:36:37.501640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.501620    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.501740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.501705    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.501852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.501830    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/7a426feb-af64-4943-ad28-71437c003d5f-kube-api-access-cr4xq\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.603423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.603394    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/7a426feb-af64-4943-ad28-71437c003d5f-kube-api-access-cr4xq\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.603587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.603447    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.603587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.603515    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.604449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.604420    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.605843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.605825    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.667488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.667409    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/7a426feb-af64-4943-ad28-71437c003d5f-kube-api-access-cr4xq\") pod \"maas-api-revoke-sub-0b7e8edb-l4pwf\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:37.703092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:37.703059    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:38.081104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:38.081076    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"]
Oct 30 22:36:38.082344 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:36:38.082319    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a426feb_af64_4943_ad28_71437c003d5f.slice/crio-8ce903a958aa6ca4f4c5e433e9e0c65305ac0c320b162deb33332b5679568e4e WatchSource:0}: Error finding container 8ce903a958aa6ca4f4c5e433e9e0c65305ac0c320b162deb33332b5679568e4e: Status 404 returned error can't find the container with id 8ce903a958aa6ca4f4c5e433e9e0c65305ac0c320b162deb33332b5679568e4e
Oct 30 22:36:38.438808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:38.438775    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf" event={"ID":"7a426feb-af64-4943-ad28-71437c003d5f","Type":"ContainerStarted","Data":"76500d8abd37ce5ccf0329c4fe56330ea87169e2b7ed60eef655083b0bc3b380"}
Oct 30 22:36:38.439248 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:38.438832    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf" event={"ID":"7a426feb-af64-4943-ad28-71437c003d5f","Type":"ContainerStarted","Data":"8ce903a958aa6ca4f4c5e433e9e0c65305ac0c320b162deb33332b5679568e4e"}
Oct 30 22:36:38.562009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:38.561955    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf" podStartSLOduration=1.561941491 podStartE2EDuration="1.561941491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:38.559427657 +0000 UTC m=+2810.083570423" watchObservedRunningTime="2026-10-30 22:36:38.561941491 +0000 UTC m=+2810.086084254"
Oct 30 22:36:39.447543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:39.447512    5031 generic.go:356] "Generic (PLEG): container finished" podID="7a426feb-af64-4943-ad28-71437c003d5f" containerID="76500d8abd37ce5ccf0329c4fe56330ea87169e2b7ed60eef655083b0bc3b380" exitCode=0
Oct 30 22:36:39.447954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:39.447591    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf" event={"ID":"7a426feb-af64-4943-ad28-71437c003d5f","Type":"ContainerDied","Data":"76500d8abd37ce5ccf0329c4fe56330ea87169e2b7ed60eef655083b0bc3b380"}
Oct 30 22:36:40.606374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.606348    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:40.632597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.632569    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a426feb-af64-4943-ad28-71437c003d5f-kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/7a426feb-af64-4943-ad28-71437c003d5f-kube-api-access-cr4xq\") pod \"7a426feb-af64-4943-ad28-71437c003d5f\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") "
Oct 30 22:36:40.632810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.632633    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-xks-ca\") pod \"7a426feb-af64-4943-ad28-71437c003d5f\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") "
Oct 30 22:36:40.632810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.632675    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-openshift-ca\") pod \"7a426feb-af64-4943-ad28-71437c003d5f\" (UID: \"7a426feb-af64-4943-ad28-71437c003d5f\") "
Oct 30 22:36:40.633135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.633106    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-openshift-ca" pod "7a426feb-af64-4943-ad28-71437c003d5f" (UID: "7a426feb-af64-4943-ad28-71437c003d5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:40.634904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.634874    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-xks-ca" pod "7a426feb-af64-4943-ad28-71437c003d5f" (UID: "7a426feb-af64-4943-ad28-71437c003d5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:40.635038 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.635011    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a426feb-af64-4943-ad28-71437c003d5f-kube-api-access-cr4xq" pod "7a426feb-af64-4943-ad28-71437c003d5f" (UID: "7a426feb-af64-4943-ad28-71437c003d5f"). InnerVolumeSpecName "kube-api-access-cr4xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:40.734422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.734344    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/7a426feb-af64-4943-ad28-71437c003d5f-kube-api-access-cr4xq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:40.734422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.734371    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:40.734422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:40.734381    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a426feb-af64-4943-ad28-71437c003d5f-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:41.468845 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:41.468818    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"
Oct 30 22:36:41.469035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:41.468853    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf" event={"ID":"7a426feb-af64-4943-ad28-71437c003d5f","Type":"ContainerDied","Data":"8ce903a958aa6ca4f4c5e433e9e0c65305ac0c320b162deb33332b5679568e4e"}
Oct 30 22:36:41.469035 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:41.468911    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce903a958aa6ca4f4c5e433e9e0c65305ac0c320b162deb33332b5679568e4e"
Oct 30 22:36:45.119855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:45.119815    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:45.363250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:45.363190    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:48.787115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.787077    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"]
Oct 30 22:36:48.787814 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.787792    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7a426feb-af64-4943-ad28-71437c003d5f" containerName="revoke-keys"
Oct 30 22:36:48.787890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.787817    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a426feb-af64-4943-ad28-71437c003d5f" containerName="revoke-keys"
Oct 30 22:36:48.788001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.787987    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a426feb-af64-4943-ad28-71437c003d5f" containerName="revoke-keys"
Oct 30 22:36:48.791644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.791626    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:48.848769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.848726    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"]
Oct 30 22:36:48.895536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.895500    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:36:48.895536 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.895507    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:36:48.895759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.895700    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:36:48.895893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.895879    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:36:48.919680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.919647    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:48.919866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.919724    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:48.919866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:48.919814    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbmv\" (UniqueName: \"kubernetes.io/projected/6c8d55f6-1175-475b-bce2-8d068d6b3a86-kube-api-access-flbmv\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.020951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.020911    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.021144 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.020983    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.021144 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.021014    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbmv\" (UniqueName: \"kubernetes.io/projected/6c8d55f6-1175-475b-bce2-8d068d6b3a86-kube-api-access-flbmv\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.021834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.021749    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.023593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.023565    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.085180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.085099    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbmv\" (UniqueName: \"kubernetes.io/projected/6c8d55f6-1175-475b-bce2-8d068d6b3a86-kube-api-access-flbmv\") pod \"maas-api-revoke-sub-3d4156f1-6nqw6\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.108801 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.108769    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:49.489594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.489567    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"]
Oct 30 22:36:49.491632 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:36:49.491591    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8d55f6_1175_475b_bce2_8d068d6b3a86.slice/crio-fb2689e084a9cc395e44740ad63445c90cce3d5708e88d758664dc6eabe77718 WatchSource:0}: Error finding container fb2689e084a9cc395e44740ad63445c90cce3d5708e88d758664dc6eabe77718: Status 404 returned error can't find the container with id fb2689e084a9cc395e44740ad63445c90cce3d5708e88d758664dc6eabe77718
Oct 30 22:36:49.520024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:49.519997    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6" event={"ID":"6c8d55f6-1175-475b-bce2-8d068d6b3a86","Type":"ContainerStarted","Data":"fb2689e084a9cc395e44740ad63445c90cce3d5708e88d758664dc6eabe77718"}
Oct 30 22:36:50.248177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:50.248089    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:36:50.528943 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:50.528908    5031 generic.go:356] "Generic (PLEG): container finished" podID="6c8d55f6-1175-475b-bce2-8d068d6b3a86" containerID="596da9c52813bd19459cabedef8f78761c0d9ff20cfe3a588b4a9930513a3a55" exitCode=0
Oct 30 22:36:50.529151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:50.528986    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6" event={"ID":"6c8d55f6-1175-475b-bce2-8d068d6b3a86","Type":"ContainerDied","Data":"596da9c52813bd19459cabedef8f78761c0d9ff20cfe3a588b4a9930513a3a55"}
Oct 30 22:36:50.542885 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:36:50.542847    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:36:51.889251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:51.877336    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:36:51.889251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:51.877405    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:36:51.992970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:51.992640    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:52.095592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.094708    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8d55f6-1175-475b-bce2-8d068d6b3a86-kube-api-access-flbmv\" (UniqueName: \"kubernetes.io/projected/6c8d55f6-1175-475b-bce2-8d068d6b3a86-kube-api-access-flbmv\") pod \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") "
Oct 30 22:36:52.095592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.094788    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-xks-ca\") pod \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") "
Oct 30 22:36:52.095592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.094856    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-openshift-ca\") pod \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\" (UID: \"6c8d55f6-1175-475b-bce2-8d068d6b3a86\") "
Oct 30 22:36:52.096451 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.096108    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-openshift-ca" pod "6c8d55f6-1175-475b-bce2-8d068d6b3a86" (UID: "6c8d55f6-1175-475b-bce2-8d068d6b3a86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:52.103689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.103649    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8d55f6-1175-475b-bce2-8d068d6b3a86-kube-api-access-flbmv" pod "6c8d55f6-1175-475b-bce2-8d068d6b3a86" (UID: "6c8d55f6-1175-475b-bce2-8d068d6b3a86"). InnerVolumeSpecName "kube-api-access-flbmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:52.103880 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.103857    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-xks-ca" pod "6c8d55f6-1175-475b-bce2-8d068d6b3a86" (UID: "6c8d55f6-1175-475b-bce2-8d068d6b3a86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:52.214193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.214098    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:52.215006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.214984    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbmv\" (UniqueName: \"kubernetes.io/projected/6c8d55f6-1175-475b-bce2-8d068d6b3a86-kube-api-access-flbmv\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:52.215160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.215144    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8d55f6-1175-475b-bce2-8d068d6b3a86-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:52.356521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.356483    5031 scope.go:122] "RemoveContainer" containerID="876a94dffc15de8436291b4ae0bb9edf559cc297dc0ac8696ec39af1b1e16843"
Oct 30 22:36:52.381743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.381717    5031 scope.go:122] "RemoveContainer" containerID="55db83119facd6cb773cfecba2479088ec2c1c00770a915f6066698883255254"
Oct 30 22:36:52.561022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.560937    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"
Oct 30 22:36:52.561022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.560969    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6" event={"ID":"6c8d55f6-1175-475b-bce2-8d068d6b3a86","Type":"ContainerDied","Data":"fb2689e084a9cc395e44740ad63445c90cce3d5708e88d758664dc6eabe77718"}
Oct 30 22:36:52.561292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:52.561039    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2689e084a9cc395e44740ad63445c90cce3d5708e88d758664dc6eabe77718"
Oct 30 22:36:55.249491 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:55.249461    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:55.563390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:36:55.563310    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:03.547052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:03.546967    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:37:06.153181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:06.153142    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:06.386633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:06.386597    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:12.221237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:12.216337    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kkskg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:37:12.221237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:12.216428    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podUID="de2775d7-88bd-440d-8034-5a7412a22128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:37:14.870951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:14.870912    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:37:20.541234 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:37:20.541173    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:37:22.635960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.635923    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"]
Oct 30 22:37:22.686269 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.686235    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nqw6"]
Oct 30 22:37:22.762709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.762673    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"]
Oct 30 22:37:22.763150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.763123    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6c8d55f6-1175-475b-bce2-8d068d6b3a86" containerName="revoke-keys"
Oct 30 22:37:22.763150 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.763141    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8d55f6-1175-475b-bce2-8d068d6b3a86" containerName="revoke-keys"
Oct 30 22:37:22.763398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.763302    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8d55f6-1175-475b-bce2-8d068d6b3a86" containerName="revoke-keys"
Oct 30 22:37:22.766864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.766843    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:22.822757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.822724    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:22.822989 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.822926    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:37:22.823089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.823049    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:22.823168 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.823152    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:22.825916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.825890    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"]
Oct 30 22:37:22.904552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.904465    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:22.904552 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.904505    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9tl\" (UniqueName: \"kubernetes.io/projected/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-kube-api-access-6j9tl\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:22.904751 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:22.904677    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.005991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.005940    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.006196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.006012    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.006196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.006045    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9tl\" (UniqueName: \"kubernetes.io/projected/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-kube-api-access-6j9tl\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.006808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.006779    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.008643 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.008618    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.070272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.070241    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9tl\" (UniqueName: \"kubernetes.io/projected/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-kube-api-access-6j9tl\") pod \"maas-api-revoke-sub-3d4156f1-b77c4\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.080067 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.080041    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:23.463043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.463017    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"]
Oct 30 22:37:23.464126 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:37:23.464099    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbb4764_fa60_46c4_9bc7_334e0d4ffb88.slice/crio-e843369735823777163a0bad9a1a3697a2d8c21edd0302f0d30e6689c196f61b WatchSource:0}: Error finding container e843369735823777163a0bad9a1a3697a2d8c21edd0302f0d30e6689c196f61b: Status 404 returned error can't find the container with id e843369735823777163a0bad9a1a3697a2d8c21edd0302f0d30e6689c196f61b
Oct 30 22:37:23.822512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.822472    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4" event={"ID":"afbb4764-fa60-46c4-9bc7-334e0d4ffb88","Type":"ContainerStarted","Data":"70c2c72ae3efd31b37aced883b6eb48464733a77738c469dec10ae4118e40a07"}
Oct 30 22:37:23.822926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.822533    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4" event={"ID":"afbb4764-fa60-46c4-9bc7-334e0d4ffb88","Type":"ContainerStarted","Data":"e843369735823777163a0bad9a1a3697a2d8c21edd0302f0d30e6689c196f61b"}
Oct 30 22:37:23.944378 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:23.944324    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4" podStartSLOduration=1.944309183 podStartE2EDuration="1.944309183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:23.941604669 +0000 UTC m=+2855.465747451" watchObservedRunningTime="2026-10-30 22:37:23.944309183 +0000 UTC m=+2855.468451969"
Oct 30 22:37:24.645232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:24.645177    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8d55f6-1175-475b-bce2-8d068d6b3a86" path="/var/lib/kubelet/pods/6c8d55f6-1175-475b-bce2-8d068d6b3a86/volumes"
Oct 30 22:37:24.830153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:24.830119    5031 generic.go:356] "Generic (PLEG): container finished" podID="afbb4764-fa60-46c4-9bc7-334e0d4ffb88" containerID="70c2c72ae3efd31b37aced883b6eb48464733a77738c469dec10ae4118e40a07" exitCode=0
Oct 30 22:37:24.830573 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:24.830236    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4" event={"ID":"afbb4764-fa60-46c4-9bc7-334e0d4ffb88","Type":"ContainerDied","Data":"70c2c72ae3efd31b37aced883b6eb48464733a77738c469dec10ae4118e40a07"}
Oct 30 22:37:25.976853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:25.976831    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:26.036546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.036523    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-openshift-ca\") pod \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") "
Oct 30 22:37:26.036700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.036585    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-xks-ca\") pod \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") "
Oct 30 22:37:26.036700 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.036648    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-kube-api-access-6j9tl\" (UniqueName: \"kubernetes.io/projected/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-kube-api-access-6j9tl\") pod \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\" (UID: \"afbb4764-fa60-46c4-9bc7-334e0d4ffb88\") "
Oct 30 22:37:26.036852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.036829    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-openshift-ca" pod "afbb4764-fa60-46c4-9bc7-334e0d4ffb88" (UID: "afbb4764-fa60-46c4-9bc7-334e0d4ffb88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:26.036963 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.036948    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:26.038851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.038827    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-xks-ca" pod "afbb4764-fa60-46c4-9bc7-334e0d4ffb88" (UID: "afbb4764-fa60-46c4-9bc7-334e0d4ffb88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:26.038940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.038879    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-kube-api-access-6j9tl" pod "afbb4764-fa60-46c4-9bc7-334e0d4ffb88" (UID: "afbb4764-fa60-46c4-9bc7-334e0d4ffb88"). InnerVolumeSpecName "kube-api-access-6j9tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:26.138271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.138186    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:26.138271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.138241    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9tl\" (UniqueName: \"kubernetes.io/projected/afbb4764-fa60-46c4-9bc7-334e0d4ffb88-kube-api-access-6j9tl\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:26.843413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.843373    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4" event={"ID":"afbb4764-fa60-46c4-9bc7-334e0d4ffb88","Type":"ContainerDied","Data":"e843369735823777163a0bad9a1a3697a2d8c21edd0302f0d30e6689c196f61b"}
Oct 30 22:37:26.843413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.843397    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"
Oct 30 22:37:26.843644 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:26.843449    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e843369735823777163a0bad9a1a3697a2d8c21edd0302f0d30e6689c196f61b"
Oct 30 22:37:37.481432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:37.481391    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:37:47.991778 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:47.991738    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:37:50.540312 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:37:50.540272    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:37:52.430931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:52.430899    5031 scope.go:122] "RemoveContainer" containerID="119a727976a35ca15c126b0bff5fb803e878267a69df334ac164526c061ecce4"
Oct 30 22:37:55.622555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:37:55.622523    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:06.240766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:06.240729    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:38:16.487836 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.487798    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"]
Oct 30 22:38:16.488492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.488476    5031 cpu_manager.go:414] "containerMap: removing container" podUID="afbb4764-fa60-46c4-9bc7-334e0d4ffb88" containerName="revoke-keys"
Oct 30 22:38:16.488551 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.488495    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbb4764-fa60-46c4-9bc7-334e0d4ffb88" containerName="revoke-keys"
Oct 30 22:38:16.488652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.488641    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbb4764-fa60-46c4-9bc7-334e0d4ffb88" containerName="revoke-keys"
Oct 30 22:38:16.492192 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.492167    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.549970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.549944    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"]
Oct 30 22:38:16.553597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.553569    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.553740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.553723    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.553882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.553859    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpsnj\" (UniqueName: \"kubernetes.io/projected/ad975053-07fb-40fa-a617-f00b988d0398-kube-api-access-vpsnj\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.596825 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.596801    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:38:16.597017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.596906    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:38:16.597017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.596902    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:38:16.597017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.597007    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:38:16.655175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.655140    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.655361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.655283    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpsnj\" (UniqueName: \"kubernetes.io/projected/ad975053-07fb-40fa-a617-f00b988d0398-kube-api-access-vpsnj\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.655361 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.655315    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.655780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.655760    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.657772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.657753    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.719111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.719082    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpsnj\" (UniqueName: \"kubernetes.io/projected/ad975053-07fb-40fa-a617-f00b988d0398-kube-api-access-vpsnj\") pod \"maas-api-revoke-sub-e2fc9c5b-s7ngc\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:16.805926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:16.805872    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:17.189316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:17.189292    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"]
Oct 30 22:38:17.191424 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:38:17.191395    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad975053_07fb_40fa_a617_f00b988d0398.slice/crio-25cb2b619f12996d8d72687776487db3a2cb4957c710becfba90655b3d0f382a WatchSource:0}: Error finding container 25cb2b619f12996d8d72687776487db3a2cb4957c710becfba90655b3d0f382a: Status 404 returned error can't find the container with id 25cb2b619f12996d8d72687776487db3a2cb4957c710becfba90655b3d0f382a
Oct 30 22:38:17.203514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:17.203486    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc" event={"ID":"ad975053-07fb-40fa-a617-f00b988d0398","Type":"ContainerStarted","Data":"25cb2b619f12996d8d72687776487db3a2cb4957c710becfba90655b3d0f382a"}
Oct 30 22:38:18.210984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:18.210948    5031 generic.go:356] "Generic (PLEG): container finished" podID="ad975053-07fb-40fa-a617-f00b988d0398" containerID="52ccddd6c06f3e794834af0f1a618b272ccf7ae5c2418e98e962bbef7295204c" exitCode=0
Oct 30 22:38:18.211402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:18.211042    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc" event={"ID":"ad975053-07fb-40fa-a617-f00b988d0398","Type":"ContainerDied","Data":"52ccddd6c06f3e794834af0f1a618b272ccf7ae5c2418e98e962bbef7295204c"}
Oct 30 22:38:19.362592 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.362567    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:19.383915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.383886    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-xks-ca\") pod \"ad975053-07fb-40fa-a617-f00b988d0398\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") "
Oct 30 22:38:19.384046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.383963    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad975053-07fb-40fa-a617-f00b988d0398-kube-api-access-vpsnj\" (UniqueName: \"kubernetes.io/projected/ad975053-07fb-40fa-a617-f00b988d0398-kube-api-access-vpsnj\") pod \"ad975053-07fb-40fa-a617-f00b988d0398\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") "
Oct 30 22:38:19.384097 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.384077    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-openshift-ca\") pod \"ad975053-07fb-40fa-a617-f00b988d0398\" (UID: \"ad975053-07fb-40fa-a617-f00b988d0398\") "
Oct 30 22:38:19.384472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.384441    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-openshift-ca" pod "ad975053-07fb-40fa-a617-f00b988d0398" (UID: "ad975053-07fb-40fa-a617-f00b988d0398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:19.386135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.386099    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-xks-ca" pod "ad975053-07fb-40fa-a617-f00b988d0398" (UID: "ad975053-07fb-40fa-a617-f00b988d0398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:19.386135 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.386116    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad975053-07fb-40fa-a617-f00b988d0398-kube-api-access-vpsnj" pod "ad975053-07fb-40fa-a617-f00b988d0398" (UID: "ad975053-07fb-40fa-a617-f00b988d0398"). InnerVolumeSpecName "kube-api-access-vpsnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:19.485830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.485762    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.485830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.485795    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpsnj\" (UniqueName: \"kubernetes.io/projected/ad975053-07fb-40fa-a617-f00b988d0398-kube-api-access-vpsnj\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.485830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.485810    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad975053-07fb-40fa-a617-f00b988d0398-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.928406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:19.928377    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:20.223726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:20.223648    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"
Oct 30 22:38:20.223726 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:20.223670    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc" event={"ID":"ad975053-07fb-40fa-a617-f00b988d0398","Type":"ContainerDied","Data":"25cb2b619f12996d8d72687776487db3a2cb4957c710becfba90655b3d0f382a"}
Oct 30 22:38:20.223927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:20.223740    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25cb2b619f12996d8d72687776487db3a2cb4957c710becfba90655b3d0f382a"
Oct 30 22:38:20.543252 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:38:20.543140    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:38:25.431981 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:25.431948    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:25.689824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:25.689747    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:27.295714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.295682    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"]
Oct 30 22:38:27.296247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.296228    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ad975053-07fb-40fa-a617-f00b988d0398" containerName="revoke-keys"
Oct 30 22:38:27.296247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.296241    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad975053-07fb-40fa-a617-f00b988d0398" containerName="revoke-keys"
Oct 30 22:38:27.296367 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.296356    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad975053-07fb-40fa-a617-f00b988d0398" containerName="revoke-keys"
Oct 30 22:38:27.299982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.299961    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.360365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.360287    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"]
Oct 30 22:38:27.361156 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.361135    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt44m\" (UniqueName: \"kubernetes.io/projected/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-kube-api-access-jt44m\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.361261 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.361197    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.361350 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.361333    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.406934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.406909    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:38:27.406934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.406921    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:38:27.407132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.407114    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:38:27.407201 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.407184    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:38:27.461848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.461824    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.461972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.461900    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt44m\" (UniqueName: \"kubernetes.io/projected/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-kube-api-access-jt44m\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.462015 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.461984    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.462612 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.462595    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.464543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.464522    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.526108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.526085    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt44m\" (UniqueName: \"kubernetes.io/projected/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-kube-api-access-jt44m\") pod \"maas-api-revoke-sub-5a29efb3-zrsff\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.614045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.614022    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:27.995349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:27.995317    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"]
Oct 30 22:38:27.996069 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:38:27.996040    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf224adc9_c0bb_4184_8938_d6f9cbd01c9b.slice/crio-2d045548e51986011c9833d29b4ca36a724d25403b94b86fc04bbeee81c00775 WatchSource:0}: Error finding container 2d045548e51986011c9833d29b4ca36a724d25403b94b86fc04bbeee81c00775: Status 404 returned error can't find the container with id 2d045548e51986011c9833d29b4ca36a724d25403b94b86fc04bbeee81c00775
Oct 30 22:38:28.274968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:28.274881    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff" event={"ID":"f224adc9-c0bb-4184-8938-d6f9cbd01c9b","Type":"ContainerStarted","Data":"2d045548e51986011c9833d29b4ca36a724d25403b94b86fc04bbeee81c00775"}
Oct 30 22:38:29.282525 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:29.282443    5031 generic.go:356] "Generic (PLEG): container finished" podID="f224adc9-c0bb-4184-8938-d6f9cbd01c9b" containerID="4e38f40650a5a056b73fd9b40019661ada92a1ef51e90349102e9fbb572e937b" exitCode=0
Oct 30 22:38:29.282884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:29.282521    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff" event={"ID":"f224adc9-c0bb-4184-8938-d6f9cbd01c9b","Type":"ContainerDied","Data":"4e38f40650a5a056b73fd9b40019661ada92a1ef51e90349102e9fbb572e937b"}
Oct 30 22:38:29.451750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:29.451714    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:38:30.468808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.468778    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:30.603520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.603482    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-xks-ca\") pod \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") "
Oct 30 22:38:30.603715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.603619    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-openshift-ca\") pod \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") "
Oct 30 22:38:30.603715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.603680    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-kube-api-access-jt44m\" (UniqueName: \"kubernetes.io/projected/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-kube-api-access-jt44m\") pod \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\" (UID: \"f224adc9-c0bb-4184-8938-d6f9cbd01c9b\") "
Oct 30 22:38:30.604833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.604806    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-openshift-ca" pod "f224adc9-c0bb-4184-8938-d6f9cbd01c9b" (UID: "f224adc9-c0bb-4184-8938-d6f9cbd01c9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:30.613243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.607039    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-kube-api-access-jt44m" pod "f224adc9-c0bb-4184-8938-d6f9cbd01c9b" (UID: "f224adc9-c0bb-4184-8938-d6f9cbd01c9b"). InnerVolumeSpecName "kube-api-access-jt44m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:30.613243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.607312    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-xks-ca" pod "f224adc9-c0bb-4184-8938-d6f9cbd01c9b" (UID: "f224adc9-c0bb-4184-8938-d6f9cbd01c9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:30.711233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.705981    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:30.711233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.706019    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt44m\" (UniqueName: \"kubernetes.io/projected/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-kube-api-access-jt44m\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:30.711233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:30.706037    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f224adc9-c0bb-4184-8938-d6f9cbd01c9b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.301876 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:31.301838    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"
Oct 30 22:38:31.302112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:31.301841    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff" event={"ID":"f224adc9-c0bb-4184-8938-d6f9cbd01c9b","Type":"ContainerDied","Data":"2d045548e51986011c9833d29b4ca36a724d25403b94b86fc04bbeee81c00775"}
Oct 30 22:38:31.302112 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:31.302011    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d045548e51986011c9833d29b4ca36a724d25403b94b86fc04bbeee81c00775"
Oct 30 22:38:35.060346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:35.060308    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"]
Oct 30 22:38:35.110965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:35.110934    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rj4f8"]
Oct 30 22:38:36.644753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:36.644721    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129a030b-0c4f-4cbf-9ee0-07e8583c8145" path="/var/lib/kubelet/pods/129a030b-0c4f-4cbf-9ee0-07e8583c8145/volumes"
Oct 30 22:38:40.273188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:40.273147    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:38:44.394530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.390450    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"]
Oct 30 22:38:44.436705 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.436670    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b77c4"]
Oct 30 22:38:44.506399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.506365    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"]
Oct 30 22:38:44.506949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.506932    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f224adc9-c0bb-4184-8938-d6f9cbd01c9b" containerName="revoke-keys"
Oct 30 22:38:44.507007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.506952    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f224adc9-c0bb-4184-8938-d6f9cbd01c9b" containerName="revoke-keys"
Oct 30 22:38:44.507092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.507037    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f224adc9-c0bb-4184-8938-d6f9cbd01c9b" containerName="revoke-keys"
Oct 30 22:38:44.511557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.511538    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.566785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.566761    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:38:44.567388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.567364    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:38:44.567521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.567508    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:38:44.569636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.569613    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"]
Oct 30 22:38:44.580183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.580157    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.580333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.580283    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.580407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.580335    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7j8\" (UniqueName: \"kubernetes.io/projected/54de8b78-5ea0-4f52-acc2-f46d9b778d14-kube-api-access-8d7j8\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.616088 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.616061    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:38:44.646730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.646646    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afbb4764-fa60-46c4-9bc7-334e0d4ffb88" path="/var/lib/kubelet/pods/afbb4764-fa60-46c4-9bc7-334e0d4ffb88/volumes"
Oct 30 22:38:44.681285 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.681258    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.681413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.681318    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7j8\" (UniqueName: \"kubernetes.io/projected/54de8b78-5ea0-4f52-acc2-f46d9b778d14-kube-api-access-8d7j8\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.681413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.681358    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.681932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.681913    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.683756 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.683737    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.745496 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.745473    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7j8\" (UniqueName: \"kubernetes.io/projected/54de8b78-5ea0-4f52-acc2-f46d9b778d14-kube-api-access-8d7j8\") pod \"maas-api-revoke-sub-3d4156f1-r78dl\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:44.824574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:44.824550    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:45.208255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:45.208223    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"]
Oct 30 22:38:45.209531 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:38:45.209497    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54de8b78_5ea0_4f52_acc2_f46d9b778d14.slice/crio-b277cfb2a04ab1e51867cf0a7a55089b1e16b182b1309b580f31f615e1dd3ddf WatchSource:0}: Error finding container b277cfb2a04ab1e51867cf0a7a55089b1e16b182b1309b580f31f615e1dd3ddf: Status 404 returned error can't find the container with id b277cfb2a04ab1e51867cf0a7a55089b1e16b182b1309b580f31f615e1dd3ddf
Oct 30 22:38:45.399250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:45.399187    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl" event={"ID":"54de8b78-5ea0-4f52-acc2-f46d9b778d14","Type":"ContainerStarted","Data":"b277cfb2a04ab1e51867cf0a7a55089b1e16b182b1309b580f31f615e1dd3ddf"}
Oct 30 22:38:46.406753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:46.406715    5031 generic.go:356] "Generic (PLEG): container finished" podID="54de8b78-5ea0-4f52-acc2-f46d9b778d14" containerID="d26b198d0e6ca146136aa56a15b28349ea0ac05c0f7ba7170229cea6c02af520" exitCode=0
Oct 30 22:38:46.407155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:46.406764    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl" event={"ID":"54de8b78-5ea0-4f52-acc2-f46d9b778d14","Type":"ContainerDied","Data":"d26b198d0e6ca146136aa56a15b28349ea0ac05c0f7ba7170229cea6c02af520"}
Oct 30 22:38:47.557897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.557867    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:47.605311 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.605283    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54de8b78-5ea0-4f52-acc2-f46d9b778d14-kube-api-access-8d7j8\" (UniqueName: \"kubernetes.io/projected/54de8b78-5ea0-4f52-acc2-f46d9b778d14-kube-api-access-8d7j8\") pod \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") "
Oct 30 22:38:47.605425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.605322    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-openshift-ca\") pod \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") "
Oct 30 22:38:47.605425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.605341    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-xks-ca\") pod \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\" (UID: \"54de8b78-5ea0-4f52-acc2-f46d9b778d14\") "
Oct 30 22:38:47.605701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.605671    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-openshift-ca" pod "54de8b78-5ea0-4f52-acc2-f46d9b778d14" (UID: "54de8b78-5ea0-4f52-acc2-f46d9b778d14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:47.607494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.607473    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-xks-ca" pod "54de8b78-5ea0-4f52-acc2-f46d9b778d14" (UID: "54de8b78-5ea0-4f52-acc2-f46d9b778d14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:47.607597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.607579    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54de8b78-5ea0-4f52-acc2-f46d9b778d14-kube-api-access-8d7j8" pod "54de8b78-5ea0-4f52-acc2-f46d9b778d14" (UID: "54de8b78-5ea0-4f52-acc2-f46d9b778d14"). InnerVolumeSpecName "kube-api-access-8d7j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:47.706998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.706922    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d7j8\" (UniqueName: \"kubernetes.io/projected/54de8b78-5ea0-4f52-acc2-f46d9b778d14-kube-api-access-8d7j8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:47.706998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.706957    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:47.706998 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:47.706971    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de8b78-5ea0-4f52-acc2-f46d9b778d14-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.076291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:48.076185    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"]
Oct 30 22:38:48.099430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:48.099402    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-msw42"]
Oct 30 22:38:48.423175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:48.423140    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl" event={"ID":"54de8b78-5ea0-4f52-acc2-f46d9b778d14","Type":"ContainerDied","Data":"b277cfb2a04ab1e51867cf0a7a55089b1e16b182b1309b580f31f615e1dd3ddf"}
Oct 30 22:38:48.423377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:48.423190    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b277cfb2a04ab1e51867cf0a7a55089b1e16b182b1309b580f31f615e1dd3ddf"
Oct 30 22:38:48.423377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:48.423149    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"
Oct 30 22:38:48.644728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:48.644694    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cf15e5-24ce-4e15-888f-8c1c19f153b3" path="/var/lib/kubelet/pods/a8cf15e5-24ce-4e15-888f-8c1c19f153b3/volumes"
Oct 30 22:38:50.539713 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:38:50.539675    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:38:56.004762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:56.004730    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:59.347370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:38:59.347338    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:39:05.362482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:05.362450    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:05.590172 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:05.590135    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:10.337634 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:10.337603    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:39:16.455872 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:16.455821    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:17.032901 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.032864    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"]
Oct 30 22:39:17.033419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.033405    5031 cpu_manager.go:414] "containerMap: removing container" podUID="54de8b78-5ea0-4f52-acc2-f46d9b778d14" containerName="revoke-keys"
Oct 30 22:39:17.033480 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.033421    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54de8b78-5ea0-4f52-acc2-f46d9b778d14" containerName="revoke-keys"
Oct 30 22:39:17.033517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.033508    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="54de8b78-5ea0-4f52-acc2-f46d9b778d14" containerName="revoke-keys"
Oct 30 22:39:17.036508 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.036493    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.096162 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.096138    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ca3b17-openshift-default-dockercfg-wkjll\""
Oct 30 22:39:17.121349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121316    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-envoy\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121366    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k68r\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-kube-api-access-8k68r\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121428    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istiod-ca-cert\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121507    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-podinfo\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121545    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-certs\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121568    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-socket\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121584    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-data\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121626    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-credential-socket\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.121740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.121646    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-token\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.140850 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.140751    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:17.145286 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.145259    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"]
Oct 30 22:39:17.222089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222059    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-envoy\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222107    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k68r\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-kube-api-access-8k68r\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222140    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istiod-ca-cert\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222181    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-podinfo\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222228    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-certs\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222263    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-socket\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222288    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-data\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222314    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-credential-socket\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222337    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-token\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.222676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222658    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-certs\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.223012 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.222959    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-credential-socket\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.223012 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.223006    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-data\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.223176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.223084    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-socket\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.223176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.223063    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istiod-ca-cert\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.224574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.224548    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-envoy\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.224884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.224866    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-podinfo\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.286621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.286549    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k68r\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-kube-api-access-8k68r\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.286856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.286832    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-token\") pod \"e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") " pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.349687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.349657    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:17.393669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.393640    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:17.782321 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.782297    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"]
Oct 30 22:39:17.783261 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:39:17.783239    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb80f8c7_532c_4df9_8695_00c3cef0d1ce.slice/crio-758f3a898e73a313fa534168525b27f0aa34d2f3230d6d32f76e8c5e04fc4a54 WatchSource:0}: Error finding container 758f3a898e73a313fa534168525b27f0aa34d2f3230d6d32f76e8c5e04fc4a54: Status 404 returned error can't find the container with id 758f3a898e73a313fa534168525b27f0aa34d2f3230d6d32f76e8c5e04fc4a54
Oct 30 22:39:17.785522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.785488    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:17.785630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.785551    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:17.785630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:17.785578    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:18.626087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:18.626056    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" event={"ID":"eb80f8c7-532c-4df9-8695-00c3cef0d1ce","Type":"ContainerStarted","Data":"eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda"}
Oct 30 22:39:18.626298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:18.626106    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" event={"ID":"eb80f8c7-532c-4df9-8695-00c3cef0d1ce","Type":"ContainerStarted","Data":"758f3a898e73a313fa534168525b27f0aa34d2f3230d6d32f76e8c5e04fc4a54"}
Oct 30 22:39:18.754326 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:18.754254    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" podStartSLOduration=2.7542376859999997 podStartE2EDuration="2.754237686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:18.752570568 +0000 UTC m=+2970.276713333" watchObservedRunningTime="2026-10-30 22:39:18.754237686 +0000 UTC m=+2970.278380446"
Oct 30 22:39:19.350396 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:19.350356    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:19.355570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:19.355545    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:19.632280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:19.632189    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:19.637049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:19.637011    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:20.541362 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:39:20.541327    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:39:21.916955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:21.916916    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:22.153167 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:22.153133    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:25.226164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:25.226132    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:25.451585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:25.451553    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:25.900892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:25.900853    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"]
Oct 30 22:39:25.905072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:25.905048    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:25.965300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:25.965275    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"]
Oct 30 22:39:26.003640 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.003616    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e5cad-6bce-4749-886c-c1298448f81f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:26.003827 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.003688    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwck8\" (UniqueName: \"kubernetes.io/projected/830e5cad-6bce-4749-886c-c1298448f81f-kube-api-access-vwck8\") pod \"payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:26.011970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.011951    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ca3b17\""
Oct 30 22:39:26.012105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.012089    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-63518488-dockercfg-v75zz\""
Oct 30 22:39:26.065892 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.065870    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"]
Oct 30 22:39:26.068575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.068548    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hlwrf"]
Oct 30 22:39:26.104740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.104714    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwck8\" (UniqueName: \"kubernetes.io/projected/830e5cad-6bce-4749-886c-c1298448f81f-kube-api-access-vwck8\") pod \"payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:26.104848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.104828    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e5cad-6bce-4749-886c-c1298448f81f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:26.105550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.105533    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e5cad-6bce-4749-886c-c1298448f81f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:26.168600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.168545    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwck8\" (UniqueName: \"kubernetes.io/projected/830e5cad-6bce-4749-886c-c1298448f81f-kube-api-access-vwck8\") pod \"payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:26.218771 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.218746    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:26.619696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.619671    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"]
Oct 30 22:39:26.620670 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:39:26.620643    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830e5cad_6bce_4749_886c_c1298448f81f.slice/crio-c23cefa5fa1c5e1a842e90d34737282c0ba8b0928d5343bb6874dbd444f3633b WatchSource:0}: Error finding container c23cefa5fa1c5e1a842e90d34737282c0ba8b0928d5343bb6874dbd444f3633b: Status 404 returned error can't find the container with id c23cefa5fa1c5e1a842e90d34737282c0ba8b0928d5343bb6874dbd444f3633b
Oct 30 22:39:26.644474 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.644452    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406836bb-78b6-4fd4-ba41-703aff195b31" path="/var/lib/kubelet/pods/406836bb-78b6-4fd4-ba41-703aff195b31/volumes"
Oct 30 22:39:26.674952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:26.674929    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" event={"ID":"830e5cad-6bce-4749-886c-c1298448f81f","Type":"ContainerStarted","Data":"c23cefa5fa1c5e1a842e90d34737282c0ba8b0928d5343bb6874dbd444f3633b"}
Oct 30 22:39:27.682404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:27.682361    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" event={"ID":"830e5cad-6bce-4749-886c-c1298448f81f","Type":"ContainerStarted","Data":"5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e"}
Oct 30 22:39:27.682858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:27.682517    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:27.807316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:27.807266    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" podStartSLOduration=2.807251967 podStartE2EDuration="2.807251967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:27.804148195 +0000 UTC m=+2979.328290971" watchObservedRunningTime="2026-10-30 22:39:27.807251967 +0000 UTC m=+2979.331394729"
Oct 30 22:39:28.544785 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:28.544753    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:28.773637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:28.773607    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:33.801431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:33.801396    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:36.850000 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:36.849961    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:39:37.062073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:37.062035    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"]
Oct 30 22:39:37.066022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:37.065990    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8xgfk"]
Oct 30 22:39:38.650262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:38.650200    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f52ab89-63a0-4741-bdad-689c0bce186f" path="/var/lib/kubelet/pods/1f52ab89-63a0-4741-bdad-689c0bce186f/volumes"
Oct 30 22:39:38.695280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:38.695256    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:39:39.005002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:39.004926    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:44.109046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:44.109011    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:45.803627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:45.803532    5031 generic.go:356] "Generic (PLEG): container finished" podID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerID="eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda" exitCode=0
Oct 30 22:39:45.803627 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:45.803595    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" event={"ID":"eb80f8c7-532c-4df9-8695-00c3cef0d1ce","Type":"ContainerDied","Data":"eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda"}
Oct 30 22:39:45.804085 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:45.803980    5031 scope.go:122] "RemoveContainer" containerID="eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda"
Oct 30 22:39:45.805136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:45.805110    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:45.805193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:45.805173    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:45.805251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:45.805234    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:46.652013 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:46.651975    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:46.837132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:46.837096    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" event={"ID":"eb80f8c7-532c-4df9-8695-00c3cef0d1ce","Type":"ContainerStarted","Data":"cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce"}
Oct 30 22:39:47.324848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:47.324809    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.350017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:47.349985    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:47.351193 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:47.351169    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 22:39:47.351331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:47.351259    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 22:39:47.624766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:47.624725    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:48.350397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:48.350356    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 22:39:48.350926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:48.350424    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 22:39:49.350714 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:49.350679    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 22:39:49.351125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:49.350732    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 22:39:49.632792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:49.632702    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:49.691530 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:49.691498    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:50.058867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:50.058790    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:39:50.074946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:50.074923    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:39:50.350745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:50.350716    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 22:39:50.351146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:50.350772    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 22:39:50.559269 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:39:50.559231    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:39:50.823292 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:50.822454    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:51.350572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:51.350537    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 30 22:39:51.350767 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:51.350588    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 30 22:39:52.355196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:52.355166    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:52.356412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:52.356388    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:39:52.468521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:52.468491    5031 scope.go:122] "RemoveContainer" containerID="92f082492798f05a98092a9d8bbc1606eac229006646d98787896f00d81a9092"
Oct 30 22:39:52.478185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:52.478153    5031 scope.go:122] "RemoveContainer" containerID="2556f70fe518879e5e555f3f2127ce7cf49360d32f9ab835d82030625353cce3"
Oct 30 22:39:54.918793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:54.918762    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:55.479202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:55.479160    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:39:56.767037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:56.766992    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:58.681454 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:39:58.681376    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:00.109907 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:00.109873    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:05.198024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:05.197990    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:10.521273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:10.521234    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:15.058743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:15.058705    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"]
Oct 30 22:40:15.109115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:15.109076    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8wwkq"]
Oct 30 22:40:15.626147 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:15.626110    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:16.644783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:16.644747    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e149abe5-35e5-4a37-b477-55928db55f90" path="/var/lib/kubelet/pods/e149abe5-35e5-4a37-b477-55928db55f90/volumes"
Oct 30 22:40:20.542880 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:40:20.542840    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:40:20.715581 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:20.715543    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:21.256179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:21.256139    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:40:25.831271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:25.831228    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:30.913429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:30.913394    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:35.134006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.133968    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"]
Oct 30 22:40:35.139601 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.139573    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.196672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.196638    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"]
Oct 30 22:40:35.243555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.243531    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:35.243699 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.243668    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.243746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.243725    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2f2\" (UniqueName: \"kubernetes.io/projected/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-kube-api-access-ws2f2\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.243803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.243765    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.243803 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.243767    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:35.243926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.243838    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:40:35.244790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.244775    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:35.345001 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.344974    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.345175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.345044    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2f2\" (UniqueName: \"kubernetes.io/projected/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-kube-api-access-ws2f2\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.345175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.345086    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.345622 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.345604    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.347574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.347555    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.417650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.417590    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2f2\" (UniqueName: \"kubernetes.io/projected/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-kube-api-access-ws2f2\") pod \"maas-api-revoke-sub-88eb1481-qwnhq\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.455624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.455592    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:35.836039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:35.836009    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"]
Oct 30 22:40:35.836490 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:40:35.836468    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb8daa6_18c8_4c8b_a398_6e2ba634e32a.slice/crio-90a0b965a1e2b25333af22c5cd1cf51682cfb66245e0228f2438bf0bca021462 WatchSource:0}: Error finding container 90a0b965a1e2b25333af22c5cd1cf51682cfb66245e0228f2438bf0bca021462: Status 404 returned error can't find the container with id 90a0b965a1e2b25333af22c5cd1cf51682cfb66245e0228f2438bf0bca021462
Oct 30 22:40:36.111689 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:36.111656    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:36.161190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:36.161157    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq" event={"ID":"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a","Type":"ContainerStarted","Data":"1c99681ecbbddfead5e2d5d3ca7e397171e6e9217b0c166abdcd4e0377726f5e"}
Oct 30 22:40:36.161614 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:36.161235    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq" event={"ID":"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a","Type":"ContainerStarted","Data":"90a0b965a1e2b25333af22c5cd1cf51682cfb66245e0228f2438bf0bca021462"}
Oct 30 22:40:36.283713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:36.283655    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq" podStartSLOduration=1.283639904 podStartE2EDuration="1.283639904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:36.281226324 +0000 UTC m=+3047.805369106" watchObservedRunningTime="2026-10-30 22:40:36.283639904 +0000 UTC m=+3047.807782704"
Oct 30 22:40:37.168769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:37.168732    5031 generic.go:356] "Generic (PLEG): container finished" podID="cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" containerID="1c99681ecbbddfead5e2d5d3ca7e397171e6e9217b0c166abdcd4e0377726f5e" exitCode=0
Oct 30 22:40:37.169294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:37.168819    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq" event={"ID":"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a","Type":"ContainerDied","Data":"1c99681ecbbddfead5e2d5d3ca7e397171e6e9217b0c166abdcd4e0377726f5e"}
Oct 30 22:40:38.320077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.320052    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:38.374958 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.374921    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-openshift-ca\") pod \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") "
Oct 30 22:40:38.375155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.374979    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-kube-api-access-ws2f2\" (UniqueName: \"kubernetes.io/projected/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-kube-api-access-ws2f2\") pod \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") "
Oct 30 22:40:38.375155 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.375042    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-xks-ca\") pod \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\" (UID: \"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a\") "
Oct 30 22:40:38.375413 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.375383    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-openshift-ca" pod "cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" (UID: "cbb8daa6-18c8-4c8b-a398-6e2ba634e32a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:38.377374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.377341    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-kube-api-access-ws2f2" pod "cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" (UID: "cbb8daa6-18c8-4c8b-a398-6e2ba634e32a"). InnerVolumeSpecName "kube-api-access-ws2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:38.377374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.377342    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-xks-ca" pod "cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" (UID: "cbb8daa6-18c8-4c8b-a398-6e2ba634e32a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:38.475949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.475863    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:38.475949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.475893    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2f2\" (UniqueName: \"kubernetes.io/projected/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-kube-api-access-ws2f2\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:38.475949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:38.475908    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:39.183416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:39.183389    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"
Oct 30 22:40:39.183628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:39.183422    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq" event={"ID":"cbb8daa6-18c8-4c8b-a398-6e2ba634e32a","Type":"ContainerDied","Data":"90a0b965a1e2b25333af22c5cd1cf51682cfb66245e0228f2438bf0bca021462"}
Oct 30 22:40:39.183628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:39.183475    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a0b965a1e2b25333af22c5cd1cf51682cfb66245e0228f2438bf0bca021462"
Oct 30 22:40:41.108906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:41.108868    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:44.063082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:44.063050    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"]
Oct 30 22:40:44.113158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:44.113123    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pqnzv"]
Oct 30 22:40:44.644482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:44.644451    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61423af-9f80-4851-a97d-d695d87ac0cb" path="/var/lib/kubelet/pods/d61423af-9f80-4851-a97d-d695d87ac0cb/volumes"
Oct 30 22:40:46.200789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:46.200752    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:47.127661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.127619    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:47.594055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.594022    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"]
Oct 30 22:40:47.594563 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.594549    5031 cpu_manager.go:414] "containerMap: removing container" podUID="cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" containerName="revoke-keys"
Oct 30 22:40:47.594611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.594564    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" containerName="revoke-keys"
Oct 30 22:40:47.594686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.594677    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" containerName="revoke-keys"
Oct 30 22:40:47.597896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.597880    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.657153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.657118    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"]
Oct 30 22:40:47.669341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.669307    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l62l7\" (UniqueName: \"kubernetes.io/projected/99c20156-1f3d-41a3-9c28-ef2e468bfc26-kube-api-access-l62l7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.669515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.669411    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.669609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.669586    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.703567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.703533    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:47.703567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.703562    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:47.703800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.703647    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:47.703800 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.703749    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:40:47.770754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.770725    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.770957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.770773    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l62l7\" (UniqueName: \"kubernetes.io/projected/99c20156-1f3d-41a3-9c28-ef2e468bfc26-kube-api-access-l62l7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.770957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.770837    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.771624 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.771600    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.773434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.773415    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.833897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.833862    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l62l7\" (UniqueName: \"kubernetes.io/projected/99c20156-1f3d-41a3-9c28-ef2e468bfc26-kube-api-access-l62l7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:47.910494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:47.910416    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:48.289543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:48.289506    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"]
Oct 30 22:40:48.290644 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:40:48.290618    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c20156_1f3d_41a3_9c28_ef2e468bfc26.slice/crio-33712187ad061b9e03ad2f0b415430ebef2a2a3bd7dca694a1b4613d4268ec3c WatchSource:0}: Error finding container 33712187ad061b9e03ad2f0b415430ebef2a2a3bd7dca694a1b4613d4268ec3c: Status 404 returned error can't find the container with id 33712187ad061b9e03ad2f0b415430ebef2a2a3bd7dca694a1b4613d4268ec3c
Oct 30 22:40:49.248991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:49.248961    5031 generic.go:356] "Generic (PLEG): container finished" podID="99c20156-1f3d-41a3-9c28-ef2e468bfc26" containerID="d62f03aa4cf47b19cf6e3d8045a136fcfd326c223b56a3f99e4c751bde8fbf0f" exitCode=0
Oct 30 22:40:49.249470 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:49.249048    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j" event={"ID":"99c20156-1f3d-41a3-9c28-ef2e468bfc26","Type":"ContainerDied","Data":"d62f03aa4cf47b19cf6e3d8045a136fcfd326c223b56a3f99e4c751bde8fbf0f"}
Oct 30 22:40:49.249470 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:49.249099    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j" event={"ID":"99c20156-1f3d-41a3-9c28-ef2e468bfc26","Type":"ContainerStarted","Data":"33712187ad061b9e03ad2f0b415430ebef2a2a3bd7dca694a1b4613d4268ec3c"}
Oct 30 22:40:50.261662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.261632    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:40:50.398377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.398352    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:50.505231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.505186    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c20156-1f3d-41a3-9c28-ef2e468bfc26-kube-api-access-l62l7\" (UniqueName: \"kubernetes.io/projected/99c20156-1f3d-41a3-9c28-ef2e468bfc26-kube-api-access-l62l7\") pod \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") "
Oct 30 22:40:50.505423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.505301    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-xks-ca\") pod \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") "
Oct 30 22:40:50.505423 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.505352    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-openshift-ca\") pod \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\" (UID: \"99c20156-1f3d-41a3-9c28-ef2e468bfc26\") "
Oct 30 22:40:50.505693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.505667    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-openshift-ca" pod "99c20156-1f3d-41a3-9c28-ef2e468bfc26" (UID: "99c20156-1f3d-41a3-9c28-ef2e468bfc26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:50.505798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.505782    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:50.507545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.507491    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c20156-1f3d-41a3-9c28-ef2e468bfc26-kube-api-access-l62l7" pod "99c20156-1f3d-41a3-9c28-ef2e468bfc26" (UID: "99c20156-1f3d-41a3-9c28-ef2e468bfc26"). InnerVolumeSpecName "kube-api-access-l62l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:50.507633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.507556    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-xks-ca" pod "99c20156-1f3d-41a3-9c28-ef2e468bfc26" (UID: "99c20156-1f3d-41a3-9c28-ef2e468bfc26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:50.552501 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:40:50.552456    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:40:50.606495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.606464    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l62l7\" (UniqueName: \"kubernetes.io/projected/99c20156-1f3d-41a3-9c28-ef2e468bfc26-kube-api-access-l62l7\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:50.606495 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:50.606493    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c20156-1f3d-41a3-9c28-ef2e468bfc26-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:51.265277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:51.265243    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j" event={"ID":"99c20156-1f3d-41a3-9c28-ef2e468bfc26","Type":"ContainerDied","Data":"33712187ad061b9e03ad2f0b415430ebef2a2a3bd7dca694a1b4613d4268ec3c"}
Oct 30 22:40:51.265681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:51.265284    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"
Oct 30 22:40:51.265681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:51.265292    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33712187ad061b9e03ad2f0b415430ebef2a2a3bd7dca694a1b4613d4268ec3c"
Oct 30 22:40:51.768722 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:51.768676    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:52.265591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:52.265555    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:52.506548 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:52.506521    5031 scope.go:122] "RemoveContainer" containerID="534d083fc4156249ce96cd1b2b9eb1094fdb78fbf41e320585d34dfea59650e2"
Oct 30 22:40:52.516855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:52.516828    5031 scope.go:122] "RemoveContainer" containerID="a563873edbfc151318ea595d209a30eda5333b2302d722ae483504cb186efb30"
Oct 30 22:40:56.793511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:56.793464    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:57.058033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:57.057937    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:58.110328 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:58.110297    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"]
Oct 30 22:40:58.434980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:58.434883    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:58.661074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:40:58.661040    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:01.509110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:01.509073    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:03.030637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:03.030604    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:03.110764 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:03.110712    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" podUID="830e5cad-6bce-4749-886c-c1298448f81f" containerName="payload-pre-processing" containerID="cri-o://5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e" gracePeriod=25
Oct 30 22:41:03.320469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:03.320394    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:03.643485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:03.643456    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"]
Oct 30 22:41:03.693878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:03.693847    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca3b17-7kw5j"]
Oct 30 22:41:04.089738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:04.089661    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:04.644528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:04.644497    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c20156-1f3d-41a3-9c28-ef2e468bfc26" path="/var/lib/kubelet/pods/99c20156-1f3d-41a3-9c28-ef2e468bfc26/volumes"
Oct 30 22:41:06.156556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:06.156506    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:41:08.693064 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:08.693034    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" podUID="830e5cad-6bce-4749-886c-c1298448f81f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:41:11.130789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:11.130743    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:11.368076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:11.367863    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:11.423385 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:11.423306    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"]
Oct 30 22:41:11.423564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:11.423543    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy" containerID="cri-o://cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce" gracePeriod=30
Oct 30 22:41:11.599607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:11.599572    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.047396 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.047362    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"]
Oct 30 22:41:13.097776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.097739    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r78dl"]
Oct 30 22:41:13.174555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.174516    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"]
Oct 30 22:41:13.175076 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.175055    5031 cpu_manager.go:414] "containerMap: removing container" podUID="99c20156-1f3d-41a3-9c28-ef2e468bfc26" containerName="revoke-keys"
Oct 30 22:41:13.175182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.175077    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c20156-1f3d-41a3-9c28-ef2e468bfc26" containerName="revoke-keys"
Oct 30 22:41:13.175304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.175258    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c20156-1f3d-41a3-9c28-ef2e468bfc26" containerName="revoke-keys"
Oct 30 22:41:13.180109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.180088    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.235104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.235070    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:13.235104 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.235083    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:13.235354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.235147    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:13.235354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.235332    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:41:13.236823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.236785    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.236950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.236919    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzlb\" (UniqueName: \"kubernetes.io/projected/184039f7-945f-4c45-9ec2-9c67356a59ff-kube-api-access-jlzlb\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.237022 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.237001    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.238279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.238256    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"]
Oct 30 22:41:13.338254 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.338157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.338409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.338263    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzlb\" (UniqueName: \"kubernetes.io/projected/184039f7-945f-4c45-9ec2-9c67356a59ff-kube-api-access-jlzlb\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.338409 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.338312    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.338797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.338770    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.340792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.340772    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.401846 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.401822    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzlb\" (UniqueName: \"kubernetes.io/projected/184039f7-945f-4c45-9ec2-9c67356a59ff-kube-api-access-jlzlb\") pod \"maas-api-revoke-sub-3d4156f1-lt4nr\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.492916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.492887    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:13.871307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:13.871270    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"]
Oct 30 22:41:13.872567 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:41:13.872538    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184039f7_945f_4c45_9ec2_9c67356a59ff.slice/crio-e626283a4996ab66ed737bdd650720959b29ce6f320066be65ad8c2d9263ce06 WatchSource:0}: Error finding container e626283a4996ab66ed737bdd650720959b29ce6f320066be65ad8c2d9263ce06: Status 404 returned error can't find the container with id e626283a4996ab66ed737bdd650720959b29ce6f320066be65ad8c2d9263ce06
Oct 30 22:41:14.416629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:14.416539    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr" event={"ID":"184039f7-945f-4c45-9ec2-9c67356a59ff","Type":"ContainerStarted","Data":"302e2566f7194e54e03d546272fc7ee715cdcdb2fa4b44de1277da64786ef004"}
Oct 30 22:41:14.416629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:14.416608    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr" event={"ID":"184039f7-945f-4c45-9ec2-9c67356a59ff","Type":"ContainerStarted","Data":"e626283a4996ab66ed737bdd650720959b29ce6f320066be65ad8c2d9263ce06"}
Oct 30 22:41:14.540786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:14.540727    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr" podStartSLOduration=1.540713249 podStartE2EDuration="1.540713249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:14.537125809 +0000 UTC m=+3086.061268574" watchObservedRunningTime="2026-10-30 22:41:14.540713249 +0000 UTC m=+3086.064856012"
Oct 30 22:41:14.645098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:14.645064    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54de8b78-5ea0-4f52-acc2-f46d9b778d14" path="/var/lib/kubelet/pods/54de8b78-5ea0-4f52-acc2-f46d9b778d14/volumes"
Oct 30 22:41:15.425370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:15.425338    5031 generic.go:356] "Generic (PLEG): container finished" podID="184039f7-945f-4c45-9ec2-9c67356a59ff" containerID="302e2566f7194e54e03d546272fc7ee715cdcdb2fa4b44de1277da64786ef004" exitCode=0
Oct 30 22:41:15.425834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:15.425390    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr" event={"ID":"184039f7-945f-4c45-9ec2-9c67356a59ff","Type":"ContainerDied","Data":"302e2566f7194e54e03d546272fc7ee715cdcdb2fa4b44de1277da64786ef004"}
Oct 30 22:41:16.576273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.576246    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:16.673122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.673096    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184039f7-945f-4c45-9ec2-9c67356a59ff-kube-api-access-jlzlb\" (UniqueName: \"kubernetes.io/projected/184039f7-945f-4c45-9ec2-9c67356a59ff-kube-api-access-jlzlb\") pod \"184039f7-945f-4c45-9ec2-9c67356a59ff\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") "
Oct 30 22:41:16.673307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.673151    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-xks-ca\") pod \"184039f7-945f-4c45-9ec2-9c67356a59ff\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") "
Oct 30 22:41:16.673307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.673238    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-openshift-ca\") pod \"184039f7-945f-4c45-9ec2-9c67356a59ff\" (UID: \"184039f7-945f-4c45-9ec2-9c67356a59ff\") "
Oct 30 22:41:16.673967 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.673934    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-openshift-ca" pod "184039f7-945f-4c45-9ec2-9c67356a59ff" (UID: "184039f7-945f-4c45-9ec2-9c67356a59ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:16.675424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.675396    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-xks-ca" pod "184039f7-945f-4c45-9ec2-9c67356a59ff" (UID: "184039f7-945f-4c45-9ec2-9c67356a59ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:16.675521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.675497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184039f7-945f-4c45-9ec2-9c67356a59ff-kube-api-access-jlzlb" pod "184039f7-945f-4c45-9ec2-9c67356a59ff" (UID: "184039f7-945f-4c45-9ec2-9c67356a59ff"). InnerVolumeSpecName "kube-api-access-jlzlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:16.685588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.685544    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:41:16.774514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774487    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-podinfo\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774530    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-envoy\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774662    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-token\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-token\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774734 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774705    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-data\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774735    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-socket\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774783    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-kube-api-access-8k68r\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-kube-api-access-8k68r\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774811    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istiod-ca-cert\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774840    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-certs\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774882    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-credential-socket\") pod \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\" (UID: \"eb80f8c7-532c-4df9-8695-00c3cef0d1ce\") "
Oct 30 22:41:16.774990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774963    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-data" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:16.775160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.774998    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-socket" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:16.775240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775193    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-certs" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:16.775316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775280    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-credential-socket" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:16.775486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775461    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istiod-ca-cert" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:16.775594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775578    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.775662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775601    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184039f7-945f-4c45-9ec2-9c67356a59ff-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.775662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775618    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.775662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775632    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.775662 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775647    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.775818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775661    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.775818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775675    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.775818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.775690    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlzlb\" (UniqueName: \"kubernetes.io/projected/184039f7-945f-4c45-9ec2-9c67356a59ff-kube-api-access-jlzlb\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.776942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.776918    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-envoy" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:16.776942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.776923    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-podinfo" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:16.777517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.777497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-token" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:16.777517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.777509    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-kube-api-access-8k68r" pod "eb80f8c7-532c-4df9-8695-00c3cef0d1ce" (UID: "eb80f8c7-532c-4df9-8695-00c3cef0d1ce"). InnerVolumeSpecName "kube-api-access-8k68r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:16.876765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.876735    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k68r\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-kube-api-access-8k68r\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.876765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.876762    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.876765 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.876774    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:16.877003 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:16.876782    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb80f8c7-532c-4df9-8695-00c3cef0d1ce-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:17.058540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.058456    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"]
Oct 30 22:41:17.062960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.062928    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lvj92"]
Oct 30 22:41:17.440594 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.440555    5031 generic.go:356] "Generic (PLEG): container finished" podID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerID="cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce" exitCode=0
Oct 30 22:41:17.440757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.440618    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"
Oct 30 22:41:17.440757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.440634    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" event={"ID":"eb80f8c7-532c-4df9-8695-00c3cef0d1ce","Type":"ContainerDied","Data":"cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce"}
Oct 30 22:41:17.440757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.440712    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw" event={"ID":"eb80f8c7-532c-4df9-8695-00c3cef0d1ce","Type":"ContainerDied","Data":"758f3a898e73a313fa534168525b27f0aa34d2f3230d6d32f76e8c5e04fc4a54"}
Oct 30 22:41:17.440757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.440746    5031 scope.go:122] "RemoveContainer" containerID="cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce"
Oct 30 22:41:17.448079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.447953    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"
Oct 30 22:41:17.448079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.447951    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr" event={"ID":"184039f7-945f-4c45-9ec2-9c67356a59ff","Type":"ContainerDied","Data":"e626283a4996ab66ed737bdd650720959b29ce6f320066be65ad8c2d9263ce06"}
Oct 30 22:41:17.448079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.448070    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e626283a4996ab66ed737bdd650720959b29ce6f320066be65ad8c2d9263ce06"
Oct 30 22:41:17.461324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.461301    5031 scope.go:122] "RemoveContainer" containerID="eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda"
Oct 30 22:41:17.472780 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.472763    5031 scope.go:122] "RemoveContainer" containerID="cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce"
Oct 30 22:41:17.473135 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:41:17.473106    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce\": container with ID starting with cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce not found: ID does not exist" containerID="cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce"
Oct 30 22:41:17.473255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.473143    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce"} err="failed to get container status \"cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce\": rpc error: code = NotFound desc = could not find container \"cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce\": container with ID starting with cd69af4506a8f0e9e82763a990acd04fbc60e1a65c51818e0708e58966557cce not found: ID does not exist"
Oct 30 22:41:17.473255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.473162    5031 scope.go:122] "RemoveContainer" containerID="eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda"
Oct 30 22:41:17.473501 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:41:17.473481    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda\": container with ID starting with eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda not found: ID does not exist" containerID="eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda"
Oct 30 22:41:17.473548 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.473508    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda"} err="failed to get container status \"eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda\": rpc error: code = NotFound desc = could not find container \"eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda\": container with ID starting with eef5f0874a0d9087744b762bb311264cad6323557cf2074792e3744e084cedda not found: ID does not exist"
Oct 30 22:41:17.666810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.666774    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"]
Oct 30 22:41:17.672241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:17.672220    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw"]
Oct 30 22:41:18.645718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:18.645688    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05b4c27-4862-41d2-96ce-35b1a941bb3e" path="/var/lib/kubelet/pods/b05b4c27-4862-41d2-96ce-35b1a941bb3e/volumes"
Oct 30 22:41:18.646133 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:18.646120    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" path="/var/lib/kubelet/pods/eb80f8c7-532c-4df9-8695-00c3cef0d1ce/volumes"
Oct 30 22:41:18.693582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:18.693552    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" podUID="830e5cad-6bce-4749-886c-c1298448f81f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:41:20.540369 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:41:20.540332    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:41:23.386023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.385961    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:41:23.437555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.437531    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830e5cad-6bce-4749-886c-c1298448f81f-kube-api-access-vwck8\" (UniqueName: \"kubernetes.io/projected/830e5cad-6bce-4749-886c-c1298448f81f-kube-api-access-vwck8\") pod \"830e5cad-6bce-4749-886c-c1298448f81f\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") "
Oct 30 22:41:23.437724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.437677    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830e5cad-6bce-4749-886c-c1298448f81f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e5cad-6bce-4749-886c-c1298448f81f-plugins-config-volume\") pod \"830e5cad-6bce-4749-886c-c1298448f81f\" (UID: \"830e5cad-6bce-4749-886c-c1298448f81f\") "
Oct 30 22:41:23.438050 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.438032    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830e5cad-6bce-4749-886c-c1298448f81f-plugins-config-volume" pod "830e5cad-6bce-4749-886c-c1298448f81f" (UID: "830e5cad-6bce-4749-886c-c1298448f81f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:23.439829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.439809    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830e5cad-6bce-4749-886c-c1298448f81f-kube-api-access-vwck8" pod "830e5cad-6bce-4749-886c-c1298448f81f" (UID: "830e5cad-6bce-4749-886c-c1298448f81f"). InnerVolumeSpecName "kube-api-access-vwck8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:23.487158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.487133    5031 generic.go:356] "Generic (PLEG): container finished" podID="830e5cad-6bce-4749-886c-c1298448f81f" containerID="5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e" exitCode=0
Oct 30 22:41:23.487309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.487183    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"
Oct 30 22:41:23.487309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.487243    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" event={"ID":"830e5cad-6bce-4749-886c-c1298448f81f","Type":"ContainerDied","Data":"5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e"}
Oct 30 22:41:23.487309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.487306    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2" event={"ID":"830e5cad-6bce-4749-886c-c1298448f81f","Type":"ContainerDied","Data":"c23cefa5fa1c5e1a842e90d34737282c0ba8b0928d5343bb6874dbd444f3633b"}
Oct 30 22:41:23.487455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.487336    5031 scope.go:122] "RemoveContainer" containerID="5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e"
Oct 30 22:41:23.497750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.497733    5031 scope.go:122] "RemoveContainer" containerID="5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e"
Oct 30 22:41:23.498034 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:41:23.498017    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e\": container with ID starting with 5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e not found: ID does not exist" containerID="5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e"
Oct 30 22:41:23.498086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.498042    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e"} err="failed to get container status \"5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e\": rpc error: code = NotFound desc = could not find container \"5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e\": container with ID starting with 5d82d7412f0470fcc9fbfc3ca3233d0460849efa9c2220c13ce29f86764d985e not found: ID does not exist"
Oct 30 22:41:23.538718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.538684    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e5cad-6bce-4749-886c-c1298448f81f-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:23.538718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.538709    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwck8\" (UniqueName: \"kubernetes.io/projected/830e5cad-6bce-4749-886c-c1298448f81f-kube-api-access-vwck8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:23.625017 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.624984    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"]
Oct 30 22:41:23.635332 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:23.635307    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2"]
Oct 30 22:41:24.645258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:24.645188    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830e5cad-6bce-4749-886c-c1298448f81f" path="/var/lib/kubelet/pods/830e5cad-6bce-4749-886c-c1298448f81f/volumes"
Oct 30 22:41:28.035406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:28.035324    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:41:29.836234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:29.815750    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:30.533899 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:30.533868    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:37.563944 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:37.563910    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:39.046710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:39.046671    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:41:41.062635 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:41.062553    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"]
Oct 30 22:41:41.113045 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:41.113013    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-l4pwf"]
Oct 30 22:41:42.645505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:42.645467    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a426feb-af64-4943-ad28-71437c003d5f" path="/var/lib/kubelet/pods/7a426feb-af64-4943-ad28-71437c003d5f/volumes"
Oct 30 22:41:47.970833 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:47.970796    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"]
Oct 30 22:41:47.971300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:47.971015    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6" podUID="22249053-b464-4921-bf38-544444b708c7" containerName="manager" containerID="cri-o://0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325" gracePeriod=10
Oct 30 22:41:48.238798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.238770    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 22:41:48.382664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.382630    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:48.392669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.392638    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22249053-b464-4921-bf38-544444b708c7-kube-api-access-xhp6p\" (UniqueName: \"kubernetes.io/projected/22249053-b464-4921-bf38-544444b708c7-kube-api-access-xhp6p\") pod \"22249053-b464-4921-bf38-544444b708c7\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") "
Oct 30 22:41:48.392830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.392685    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22249053-b464-4921-bf38-544444b708c7-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22249053-b464-4921-bf38-544444b708c7-extensions-socket-volume\") pod \"22249053-b464-4921-bf38-544444b708c7\" (UID: \"22249053-b464-4921-bf38-544444b708c7\") "
Oct 30 22:41:48.393103 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.393079    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22249053-b464-4921-bf38-544444b708c7-extensions-socket-volume" pod "22249053-b464-4921-bf38-544444b708c7" (UID: "22249053-b464-4921-bf38-544444b708c7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:48.395012 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.394992    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22249053-b464-4921-bf38-544444b708c7-kube-api-access-xhp6p" pod "22249053-b464-4921-bf38-544444b708c7" (UID: "22249053-b464-4921-bf38-544444b708c7"). InnerVolumeSpecName "kube-api-access-xhp6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:48.494113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.494029    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhp6p\" (UniqueName: \"kubernetes.io/projected/22249053-b464-4921-bf38-544444b708c7-kube-api-access-xhp6p\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:48.494113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.494058    5031 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22249053-b464-4921-bf38-544444b708c7-extensions-socket-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:48.702949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.702916    5031 generic.go:356] "Generic (PLEG): container finished" podID="22249053-b464-4921-bf38-544444b708c7" containerID="0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325" exitCode=0
Oct 30 22:41:48.703110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.702963    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"
Oct 30 22:41:48.703110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.702999    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6" event={"ID":"22249053-b464-4921-bf38-544444b708c7","Type":"ContainerDied","Data":"0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325"}
Oct 30 22:41:48.703110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.703045    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6" event={"ID":"22249053-b464-4921-bf38-544444b708c7","Type":"ContainerDied","Data":"eea89a4b1887d6a795924e2ee12728700fdeb6fedcf58f2bce292ccdd83f63ae"}
Oct 30 22:41:48.703110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.703074    5031 scope.go:122] "RemoveContainer" containerID="0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325"
Oct 30 22:41:48.715774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.715756    5031 scope.go:122] "RemoveContainer" containerID="0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325"
Oct 30 22:41:48.716047 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:41:48.716029    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325\": container with ID starting with 0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325 not found: ID does not exist" containerID="0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325"
Oct 30 22:41:48.716105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.716058    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325"} err="failed to get container status \"0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325\": rpc error: code = NotFound desc = could not find container \"0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325\": container with ID starting with 0fb67d1bb81065745bdb1403d3c0ebd7cbcb4bc23bb10300edb06d830dfb8325 not found: ID does not exist"
Oct 30 22:41:48.873832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.873798    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"]
Oct 30 22:41:48.911694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:48.911662    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhrv6"]
Oct 30 22:41:50.540883 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:41:50.540848    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:41:50.645305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:50.645270    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22249053-b464-4921-bf38-544444b708c7" path="/var/lib/kubelet/pods/22249053-b464-4921-bf38-544444b708c7/volumes"
Oct 30 22:41:52.546894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:52.546860    5031 scope.go:122] "RemoveContainer" containerID="44359118473f79a0fb2dde2199f7007aebfd955d51bb7ad05c4ccf89d81ce67d"
Oct 30 22:41:52.559403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:52.559378    5031 scope.go:122] "RemoveContainer" containerID="546ac43654bd78ba636f011bd71002c60ebc5da070275456335d30c34ce078ec"
Oct 30 22:41:57.037511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:57.037419    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:57.322439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:41:57.322358    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:20.540956 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:42:20.540920    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:42:50.542238 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:42:50.542178    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:42:51.613737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.613699    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"]
Oct 30 22:42:51.614188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614174    5031 cpu_manager.go:414] "containerMap: removing container" podUID="184039f7-945f-4c45-9ec2-9c67356a59ff" containerName="revoke-keys"
Oct 30 22:42:51.614266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614190    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184039f7-945f-4c45-9ec2-9c67356a59ff" containerName="revoke-keys"
Oct 30 22:42:51.614266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614239    5031 cpu_manager.go:414] "containerMap: removing container" podUID="830e5cad-6bce-4749-886c-c1298448f81f" containerName="payload-pre-processing"
Oct 30 22:42:51.614266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614246    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830e5cad-6bce-4749-886c-c1298448f81f" containerName="payload-pre-processing"
Oct 30 22:42:51.614266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614253    5031 cpu_manager.go:414] "containerMap: removing container" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy"
Oct 30 22:42:51.614266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614257    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614269    5031 cpu_manager.go:414] "containerMap: removing container" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614274    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614281    5031 cpu_manager.go:414] "containerMap: removing container" podUID="22249053-b464-4921-bf38-544444b708c7" containerName="manager"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614285    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22249053-b464-4921-bf38-544444b708c7" containerName="manager"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614366    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614375    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="22249053-b464-4921-bf38-544444b708c7" containerName="manager"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614391    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="184039f7-945f-4c45-9ec2-9c67356a59ff" containerName="revoke-keys"
Oct 30 22:42:51.614436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.614400    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="830e5cad-6bce-4749-886c-c1298448f81f" containerName="payload-pre-processing"
Oct 30 22:42:51.617576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.617559    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:51.670893 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.670858    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"]
Oct 30 22:42:51.708400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.708364    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/191e15f8-408c-4914-a9ee-f46f43bb2b6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zdsf7\" (UID: \"191e15f8-408c-4914-a9ee-f46f43bb2b6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:51.708698 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.708670    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9c9\" (UniqueName: \"kubernetes.io/projected/191e15f8-408c-4914-a9ee-f46f43bb2b6c-kube-api-access-jb9c9\") pod \"kuadrant-operator-controller-manager-678d87766f-zdsf7\" (UID: \"191e15f8-408c-4914-a9ee-f46f43bb2b6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:51.718485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.718442    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z97mv\""
Oct 30 22:42:51.810452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.810416    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9c9\" (UniqueName: \"kubernetes.io/projected/191e15f8-408c-4914-a9ee-f46f43bb2b6c-kube-api-access-jb9c9\") pod \"kuadrant-operator-controller-manager-678d87766f-zdsf7\" (UID: \"191e15f8-408c-4914-a9ee-f46f43bb2b6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:51.810636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.810495    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/191e15f8-408c-4914-a9ee-f46f43bb2b6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zdsf7\" (UID: \"191e15f8-408c-4914-a9ee-f46f43bb2b6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:51.810862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.810846    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/191e15f8-408c-4914-a9ee-f46f43bb2b6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zdsf7\" (UID: \"191e15f8-408c-4914-a9ee-f46f43bb2b6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:51.876938 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.876863    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9c9\" (UniqueName: \"kubernetes.io/projected/191e15f8-408c-4914-a9ee-f46f43bb2b6c-kube-api-access-jb9c9\") pod \"kuadrant-operator-controller-manager-678d87766f-zdsf7\" (UID: \"191e15f8-408c-4914-a9ee-f46f43bb2b6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:51.930894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:51.930856    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:52.367739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:52.367708    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"]
Oct 30 22:42:52.368276 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:42:52.368251    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191e15f8_408c_4914_a9ee_f46f43bb2b6c.slice/crio-4eab182856dca7b7b4bf3974189a8ee7e21a6b89b2cc84b63271bca69dbce335 WatchSource:0}: Error finding container 4eab182856dca7b7b4bf3974189a8ee7e21a6b89b2cc84b63271bca69dbce335: Status 404 returned error can't find the container with id 4eab182856dca7b7b4bf3974189a8ee7e21a6b89b2cc84b63271bca69dbce335
Oct 30 22:42:52.600914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:52.600863    5031 scope.go:122] "RemoveContainer" containerID="a2f5575caafa1b186183c8afabd9f3f0114fc6c791d312fa8271cdb5a229ed14"
Oct 30 22:42:52.610515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:52.610495    5031 scope.go:122] "RemoveContainer" containerID="76500d8abd37ce5ccf0329c4fe56330ea87169e2b7ed60eef655083b0bc3b380"
Oct 30 22:42:52.619433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:52.619415    5031 scope.go:122] "RemoveContainer" containerID="596da9c52813bd19459cabedef8f78761c0d9ff20cfe3a588b4a9930513a3a55"
Oct 30 22:42:53.089036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:53.088935    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7" event={"ID":"191e15f8-408c-4914-a9ee-f46f43bb2b6c","Type":"ContainerStarted","Data":"9755139ad56a62c5fa4dd8f91ad1f4f3c41d49b1adb791dbaa417a85838716d8"}
Oct 30 22:42:53.089036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:53.089002    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7" event={"ID":"191e15f8-408c-4914-a9ee-f46f43bb2b6c","Type":"ContainerStarted","Data":"4eab182856dca7b7b4bf3974189a8ee7e21a6b89b2cc84b63271bca69dbce335"}
Oct 30 22:42:53.089301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:53.089071    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:42:53.214968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:53.214905    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7" podStartSLOduration=2.214892056 podStartE2EDuration="2.214892056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:53.212871089 +0000 UTC m=+3184.737013883" watchObservedRunningTime="2026-10-30 22:42:53.214892056 +0000 UTC m=+3184.739034820"
Oct 30 22:42:57.391002 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:42:57.390875    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:04.097011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:04.096975    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zdsf7"
Oct 30 22:43:13.563522 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:13.563487    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:43:18.078761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.078725    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"]
Oct 30 22:43:18.079287 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.079273    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb80f8c7-532c-4df9-8695-00c3cef0d1ce" containerName="istio-proxy"
Oct 30 22:43:18.082600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.082582    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.113613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.113578    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.113761 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.113634    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdxc\" (UniqueName: \"kubernetes.io/projected/523a7889-59a7-4c2a-a153-6e6afe28a933-kube-api-access-zgdxc\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.113843 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.113820    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.141436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.141384    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:18.141973 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.141451    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:18.142740 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.142704    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:18.192012 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.191988    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:43:18.194933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.194910    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"]
Oct 30 22:43:18.215180 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.215155    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.215340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.215320    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.215402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.215364    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdxc\" (UniqueName: \"kubernetes.io/projected/523a7889-59a7-4c2a-a153-6e6afe28a933-kube-api-access-zgdxc\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.215792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.215763    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.218033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.218006    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.278990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.278961    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdxc\" (UniqueName: \"kubernetes.io/projected/523a7889-59a7-4c2a-a153-6e6afe28a933-kube-api-access-zgdxc\") pod \"maas-api-revoke-sub-d576e68b-6cv7x\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.397418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.397388    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:18.779518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:18.779492    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"]
Oct 30 22:43:18.781457 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:43:18.781435    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523a7889_59a7_4c2a_a153_6e6afe28a933.slice/crio-4c57668b851881d36ea986a1c1459f70155037a2ae706fa6725cc4e7e3aff364 WatchSource:0}: Error finding container 4c57668b851881d36ea986a1c1459f70155037a2ae706fa6725cc4e7e3aff364: Status 404 returned error can't find the container with id 4c57668b851881d36ea986a1c1459f70155037a2ae706fa6725cc4e7e3aff364
Oct 30 22:43:19.283739 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:19.283699    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x" event={"ID":"523a7889-59a7-4c2a-a153-6e6afe28a933","Type":"ContainerStarted","Data":"49efb01a2b4371951da36e4693d8fa37283950bf1eff72fbfce28c9db3a21890"}
Oct 30 22:43:19.284161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:19.283761    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x" event={"ID":"523a7889-59a7-4c2a-a153-6e6afe28a933","Type":"ContainerStarted","Data":"4c57668b851881d36ea986a1c1459f70155037a2ae706fa6725cc4e7e3aff364"}
Oct 30 22:43:19.404744 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:19.404625    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x" podStartSLOduration=1.404610697 podStartE2EDuration="1.404610697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:19.403633087 +0000 UTC m=+3210.927775872" watchObservedRunningTime="2026-10-30 22:43:19.404610697 +0000 UTC m=+3210.928753506"
Oct 30 22:43:20.074336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:20.074261    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"]
Oct 30 22:43:20.122667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:20.122637    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-s7ngc"]
Oct 30 22:43:20.291125 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:20.291087    5031 generic.go:356] "Generic (PLEG): container finished" podID="523a7889-59a7-4c2a-a153-6e6afe28a933" containerID="49efb01a2b4371951da36e4693d8fa37283950bf1eff72fbfce28c9db3a21890" exitCode=0
Oct 30 22:43:20.291539 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:20.291195    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x" event={"ID":"523a7889-59a7-4c2a-a153-6e6afe28a933","Type":"ContainerDied","Data":"49efb01a2b4371951da36e4693d8fa37283950bf1eff72fbfce28c9db3a21890"}
Oct 30 22:43:20.541038 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:43:20.540999    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:43:20.644578 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:20.644542    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad975053-07fb-40fa-a617-f00b988d0398" path="/var/lib/kubelet/pods/ad975053-07fb-40fa-a617-f00b988d0398/volumes"
Oct 30 22:43:21.453305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.453275    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:21.547638 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.547576    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523a7889-59a7-4c2a-a153-6e6afe28a933-kube-api-access-zgdxc\" (UniqueName: \"kubernetes.io/projected/523a7889-59a7-4c2a-a153-6e6afe28a933-kube-api-access-zgdxc\") pod \"523a7889-59a7-4c2a-a153-6e6afe28a933\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") "
Oct 30 22:43:21.547784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.547736    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-xks-ca\") pod \"523a7889-59a7-4c2a-a153-6e6afe28a933\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") "
Oct 30 22:43:21.547784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.547760    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-openshift-ca\") pod \"523a7889-59a7-4c2a-a153-6e6afe28a933\" (UID: \"523a7889-59a7-4c2a-a153-6e6afe28a933\") "
Oct 30 22:43:21.548153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.548123    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-openshift-ca" pod "523a7889-59a7-4c2a-a153-6e6afe28a933" (UID: "523a7889-59a7-4c2a-a153-6e6afe28a933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:21.549777 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.549755    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-xks-ca" pod "523a7889-59a7-4c2a-a153-6e6afe28a933" (UID: "523a7889-59a7-4c2a-a153-6e6afe28a933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:21.549876 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.549852    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523a7889-59a7-4c2a-a153-6e6afe28a933-kube-api-access-zgdxc" pod "523a7889-59a7-4c2a-a153-6e6afe28a933" (UID: "523a7889-59a7-4c2a-a153-6e6afe28a933"). InnerVolumeSpecName "kube-api-access-zgdxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:21.648481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.648450    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:21.648481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.648477    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a7889-59a7-4c2a-a153-6e6afe28a933-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:21.648481 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:21.648486    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdxc\" (UniqueName: \"kubernetes.io/projected/523a7889-59a7-4c2a-a153-6e6afe28a933-kube-api-access-zgdxc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:22.304555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:22.304524    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"
Oct 30 22:43:22.304733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:22.304523    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x" event={"ID":"523a7889-59a7-4c2a-a153-6e6afe28a933","Type":"ContainerDied","Data":"4c57668b851881d36ea986a1c1459f70155037a2ae706fa6725cc4e7e3aff364"}
Oct 30 22:43:22.304733 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:22.304649    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c57668b851881d36ea986a1c1459f70155037a2ae706fa6725cc4e7e3aff364"
Oct 30 22:43:31.054691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:31.054653    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:43:31.105216 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:31.105180    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"]
Oct 30 22:43:31.107900 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:31.107873    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zrsff"]
Oct 30 22:43:32.644673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:32.644641    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f224adc9-c0bb-4184-8938-d6f9cbd01c9b" path="/var/lib/kubelet/pods/f224adc9-c0bb-4184-8938-d6f9cbd01c9b/volumes"
Oct 30 22:43:40.890312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:40.890271    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:43:47.278964 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.278925    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"]
Oct 30 22:43:47.279505 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.279488    5031 cpu_manager.go:414] "containerMap: removing container" podUID="523a7889-59a7-4c2a-a153-6e6afe28a933" containerName="revoke-keys"
Oct 30 22:43:47.279554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.279506    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523a7889-59a7-4c2a-a153-6e6afe28a933" containerName="revoke-keys"
Oct 30 22:43:47.279602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.279593    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="523a7889-59a7-4c2a-a153-6e6afe28a933" containerName="revoke-keys"
Oct 30 22:43:47.284078 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.284057    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.339428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.339398    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:43:47.341815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.341788    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"]
Oct 30 22:43:47.389080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.389044    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:47.389528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.389504    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:47.430602 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.430568    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.430790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.430772    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.430858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.430822    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92dq\" (UniqueName: \"kubernetes.io/projected/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-kube-api-access-j92dq\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.439306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.439283    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:47.532483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.532409    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.532483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.532445    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92dq\" (UniqueName: \"kubernetes.io/projected/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-kube-api-access-j92dq\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.532483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.532484    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.533312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.533202    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.535043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.535023    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.596412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.596382    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92dq\" (UniqueName: \"kubernetes.io/projected/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-kube-api-access-j92dq\") pod \"maas-api-revoke-sub-d2bc93bc-gskpz\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.598190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.598172    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:47.978256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:47.978227    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"]
Oct 30 22:43:47.980285 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:43:47.980259    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2f5894_0f32_4c3b_8d69_cc6fda43f44b.slice/crio-dad1f94652b7998a7d1a4d9d5debb36c976532800a89c5b90fa818d8a6e44762 WatchSource:0}: Error finding container dad1f94652b7998a7d1a4d9d5debb36c976532800a89c5b90fa818d8a6e44762: Status 404 returned error can't find the container with id dad1f94652b7998a7d1a4d9d5debb36c976532800a89c5b90fa818d8a6e44762
Oct 30 22:43:48.467608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:48.467569    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz" event={"ID":"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b","Type":"ContainerStarted","Data":"8674c5ed0b1887e95b7cc7fe0d7fdbfe05503a2c29abfd091bd59b395363f847"}
Oct 30 22:43:48.468054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:48.467632    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz" event={"ID":"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b","Type":"ContainerStarted","Data":"dad1f94652b7998a7d1a4d9d5debb36c976532800a89c5b90fa818d8a6e44762"}
Oct 30 22:43:48.590111 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:48.590027    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz" podStartSLOduration=1.59001409 podStartE2EDuration="1.59001409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:48.588200595 +0000 UTC m=+3240.112343358" watchObservedRunningTime="2026-10-30 22:43:48.59001409 +0000 UTC m=+3240.114156853"
Oct 30 22:43:49.475039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:49.475007    5031 generic.go:356] "Generic (PLEG): container finished" podID="7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" containerID="8674c5ed0b1887e95b7cc7fe0d7fdbfe05503a2c29abfd091bd59b395363f847" exitCode=0
Oct 30 22:43:49.475490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:49.475096    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz" event={"ID":"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b","Type":"ContainerDied","Data":"8674c5ed0b1887e95b7cc7fe0d7fdbfe05503a2c29abfd091bd59b395363f847"}
Oct 30 22:43:50.546271 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:43:50.546236    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:43:50.649336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.649303    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:50.762456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.762386    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-kube-api-access-j92dq\" (UniqueName: \"kubernetes.io/projected/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-kube-api-access-j92dq\") pod \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") "
Oct 30 22:43:50.762456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.762422    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-openshift-ca\") pod \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") "
Oct 30 22:43:50.762642 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.762503    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-xks-ca\") pod \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\" (UID: \"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b\") "
Oct 30 22:43:50.762820 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.762794    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-openshift-ca" pod "7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" (UID: "7a2f5894-0f32-4c3b-8d69-cc6fda43f44b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:50.764688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.764668    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-kube-api-access-j92dq" pod "7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" (UID: "7a2f5894-0f32-4c3b-8d69-cc6fda43f44b"). InnerVolumeSpecName "kube-api-access-j92dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:50.764816 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.764801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-xks-ca" pod "7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" (UID: "7a2f5894-0f32-4c3b-8d69-cc6fda43f44b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:50.863819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.863792    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92dq\" (UniqueName: \"kubernetes.io/projected/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-kube-api-access-j92dq\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:50.863819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.863816    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:50.863819 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:50.863825    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:51.491500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:51.491465    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz" event={"ID":"7a2f5894-0f32-4c3b-8d69-cc6fda43f44b","Type":"ContainerDied","Data":"dad1f94652b7998a7d1a4d9d5debb36c976532800a89c5b90fa818d8a6e44762"}
Oct 30 22:43:51.491500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:51.491501    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"
Oct 30 22:43:51.491715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:51.491524    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad1f94652b7998a7d1a4d9d5debb36c976532800a89c5b90fa818d8a6e44762"
Oct 30 22:43:52.651070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:52.651042    5031 scope.go:122] "RemoveContainer" containerID="70c2c72ae3efd31b37aced883b6eb48464733a77738c469dec10ae4118e40a07"
Oct 30 22:43:59.648829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:59.648794    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:59.901605 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:43:59.901534    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:00.254660 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.254560    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:44:00.305232 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.305174    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"]
Oct 30 22:44:00.305701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.305688    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" containerName="revoke-keys"
Oct 30 22:44:00.305748 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.305704    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" containerName="revoke-keys"
Oct 30 22:44:00.305823 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.305814    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" containerName="revoke-keys"
Oct 30 22:44:00.310478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.310451    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.349744    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-podinfo\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.349801    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-certs\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.349873    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-socket\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.349955    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-token\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.349985    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-data\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.350016    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istiod-ca-cert\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.350099    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-credential-socket\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.350179    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsxx\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-kube-api-access-sgsxx\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.356443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.350228    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-envoy\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.385236 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.385187    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ef21ee-openshift-default-dockercfg-g7cqq\""
Oct 30 22:44:00.387796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.387763    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"]
Oct 30 22:44:00.451565 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451534    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsxx\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-kube-api-access-sgsxx\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.451783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451569    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-envoy\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.451783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451624    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-podinfo\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.451783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451648    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-certs\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.451783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451692    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-socket\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.451783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451722    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-token\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.451783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451746    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-data\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.451783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451773    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istiod-ca-cert\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.452623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.451862    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-credential-socket\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.452623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.452481    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-credential-socket\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.452732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.452668    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-certs\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.452904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.452878    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istiod-ca-cert\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.452979 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.452955    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-data\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.453234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.453174    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-socket\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.454772 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.454752    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-envoy\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.455301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.455279    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-podinfo\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.477629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.477599    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:00.515184 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.515132    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-token\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.515299 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.515239    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsxx\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-kube-api-access-sgsxx\") pod \"e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") " pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.626434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.626387    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:00.714991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:00.714473    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:01.081333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.081309    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"]
Oct 30 22:44:01.082274 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:44:01.082251    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f7e8a7_16ff_47f0_9ab0_a28e60a76be5.slice/crio-1ee8782e34d08ff0fddf6dbd40c5f2e35d280238ab92ea1364b4aceb98ba4bd3 WatchSource:0}: Error finding container 1ee8782e34d08ff0fddf6dbd40c5f2e35d280238ab92ea1364b4aceb98ba4bd3: Status 404 returned error can't find the container with id 1ee8782e34d08ff0fddf6dbd40c5f2e35d280238ab92ea1364b4aceb98ba4bd3
Oct 30 22:44:01.084350 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.084315    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:01.084446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.084381    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:01.084446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.084408    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:01.565040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.565003    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" event={"ID":"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5","Type":"ContainerStarted","Data":"75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118"}
Oct 30 22:44:01.565270 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.565060    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" event={"ID":"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5","Type":"ContainerStarted","Data":"1ee8782e34d08ff0fddf6dbd40c5f2e35d280238ab92ea1364b4aceb98ba4bd3"}
Oct 30 22:44:01.627161 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.627126    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:01.691831 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:01.691783    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" podStartSLOduration=1.69177047 podStartE2EDuration="1.69177047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:01.689773859 +0000 UTC m=+3253.213916640" watchObservedRunningTime="2026-10-30 22:44:01.69177047 +0000 UTC m=+3253.215913231"
Oct 30 22:44:02.632924 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:02.632896    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:03.578781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:03.578749    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:03.579667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:03.579642    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:04.803738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:04.803711    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:05.033533 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:05.033496    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:08.071405 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.071372    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:08.299295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.299256    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:08.729521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.729487    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"]
Oct 30 22:44:08.733372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.733353    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:08.793061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.793032    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"]
Oct 30 22:44:08.839903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.839870    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c7b5f1b5-dockercfg-h2jg5\""
Oct 30 22:44:08.839903 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.839875    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ef21ee\""
Oct 30 22:44:08.841305 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.841284    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cdb\" (UniqueName: \"kubernetes.io/projected/044327ac-2d86-4265-9077-80353e48cbd4-kube-api-access-48cdb\") pod \"payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:08.841355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.841321    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044327ac-2d86-4265-9077-80353e48cbd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:08.942109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.942080    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cdb\" (UniqueName: \"kubernetes.io/projected/044327ac-2d86-4265-9077-80353e48cbd4-kube-api-access-48cdb\") pod \"payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:08.942284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.942118    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044327ac-2d86-4265-9077-80353e48cbd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:08.942862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:08.942840    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044327ac-2d86-4265-9077-80353e48cbd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:09.006113 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:09.006050    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cdb\" (UniqueName: \"kubernetes.io/projected/044327ac-2d86-4265-9077-80353e48cbd4-kube-api-access-48cdb\") pod \"payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:09.046968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:09.046939    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:09.423774 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:09.423744    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"]
Oct 30 22:44:09.425103 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:44:09.425078    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044327ac_2d86_4265_9077_80353e48cbd4.slice/crio-8ce556de3c8dd607d4a5672e4aa35c6d55d5124c84f975914a0b423487c16f48 WatchSource:0}: Error finding container 8ce556de3c8dd607d4a5672e4aa35c6d55d5124c84f975914a0b423487c16f48: Status 404 returned error can't find the container with id 8ce556de3c8dd607d4a5672e4aa35c6d55d5124c84f975914a0b423487c16f48
Oct 30 22:44:09.616483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:09.616445    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" event={"ID":"044327ac-2d86-4265-9077-80353e48cbd4","Type":"ContainerStarted","Data":"81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84"}
Oct 30 22:44:09.616648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:09.616506    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" event={"ID":"044327ac-2d86-4265-9077-80353e48cbd4","Type":"ContainerStarted","Data":"8ce556de3c8dd607d4a5672e4aa35c6d55d5124c84f975914a0b423487c16f48"}
Oct 30 22:44:09.616648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:09.616543    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:09.739543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:09.739446    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" podStartSLOduration=1.739433078 podStartE2EDuration="1.739433078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:09.73702473 +0000 UTC m=+3261.261167512" watchObservedRunningTime="2026-10-30 22:44:09.739433078 +0000 UTC m=+3261.263575841"
Oct 30 22:44:11.382490 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:11.382456    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:11.620695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:11.620663    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:11.842826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:11.842798    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:16.239433 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:16.239375    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:16.633822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:16.633792    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:19.578422 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:19.578386    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:44:20.561813 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:44:20.561776    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:44:20.645798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:20.645768    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:44:21.781026 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:21.780967    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:26.800460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:26.800427    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:29.624036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:29.624000    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:29.762044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:29.762012    5031 generic.go:356] "Generic (PLEG): container finished" podID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerID="75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118" exitCode=0
Oct 30 22:44:29.762238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:29.762085    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" event={"ID":"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5","Type":"ContainerDied","Data":"75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118"}
Oct 30 22:44:29.762500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:29.762480    5031 scope.go:122] "RemoveContainer" containerID="75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118"
Oct 30 22:44:29.763677 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:29.763649    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:29.763752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:29.763719    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:29.763752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:29.763748    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:30.223821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:30.223773    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:30.472189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:30.472149    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:30.769812 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:30.769779    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" event={"ID":"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5","Type":"ContainerStarted","Data":"e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b"}
Oct 30 22:44:31.628529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:31.628488    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:44:31.628832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:31.628801    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:44:31.629330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:31.629309    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:32.249858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:32.249823    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:32.631293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:32.628265    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:44:32.631293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:32.628325    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:44:33.578829 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:33.578799    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:33.627695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:33.627654    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:44:33.629019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:33.628987    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:44:33.745330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:33.744842    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:34.631663 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:34.631621    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:44:34.632173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:34.631685    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:44:35.627965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:35.627932    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:44:35.628186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:35.628000    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:44:36.631411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:36.631381    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:36.632488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:36.632465    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:44:37.319858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:37.319815    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:38.426755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:38.426721    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:44:42.525241 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:42.525178    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:47.598639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:47.598591    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:50.101004 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:50.100970    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:44:50.118478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:50.118457    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:44:50.540928 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:44:50.540839    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:44:52.677345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:52.677309    5031 scope.go:122] "RemoveContainer" containerID="4e38f40650a5a056b73fd9b40019661ada92a1ef51e90349102e9fbb572e937b"
Oct 30 22:44:52.694187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:52.694153    5031 scope.go:122] "RemoveContainer" containerID="52ccddd6c06f3e794834af0f1a618b272ccf7ae5c2418e98e962bbef7295204c"
Oct 30 22:44:52.706382 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:52.706352    5031 scope.go:122] "RemoveContainer" containerID="d26b198d0e6ca146136aa56a15b28349ea0ac05c0f7ba7170229cea6c02af520"
Oct 30 22:44:52.736568 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:52.736543    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:57.720429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:57.720392    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:58.091044 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:58.090952    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:58.373617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:44:58.373583    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:00.170688 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.170654    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"]
Oct 30 22:45:00.175876 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.175856    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"]
Oct 30 22:45:00.176006 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.175965    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.179439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.179423    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.231080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.231051    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:00.231080 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.231072    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:00.280575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.280538    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:45:00.285072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.285044    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"]
Oct 30 22:45:00.289291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.289269    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"]
Oct 30 22:45:00.310557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.310532    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2ffc65-9dae-459d-8cae-e337d19522a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.310679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.310655    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggw5q\" (UniqueName: \"kubernetes.io/projected/a2e826a4-e256-43ad-8591-7812fa30010b-kube-api-access-ggw5q\") pod \"maas-api-key-cleanup-29890005-vntzg\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.310746 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.310691    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hzzm\" (UniqueName: \"kubernetes.io/projected/ce2ffc65-9dae-459d-8cae-e337d19522a2-kube-api-access-9hzzm\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.310796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.310772    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e826a4-e256-43ad-8591-7812fa30010b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-vntzg\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.411355 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.411315    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2ffc65-9dae-459d-8cae-e337d19522a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.411564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.411446    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggw5q\" (UniqueName: \"kubernetes.io/projected/a2e826a4-e256-43ad-8591-7812fa30010b-kube-api-access-ggw5q\") pod \"maas-api-key-cleanup-29890005-vntzg\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.411564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.411478    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hzzm\" (UniqueName: \"kubernetes.io/projected/ce2ffc65-9dae-459d-8cae-e337d19522a2-kube-api-access-9hzzm\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.411564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.411504    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e826a4-e256-43ad-8591-7812fa30010b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-vntzg\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.412086 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.412058    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2ffc65-9dae-459d-8cae-e337d19522a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.412230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.412185    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e826a4-e256-43ad-8591-7812fa30010b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-vntzg\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.475316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.475244    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggw5q\" (UniqueName: \"kubernetes.io/projected/a2e826a4-e256-43ad-8591-7812fa30010b-kube-api-access-ggw5q\") pod \"maas-api-key-cleanup-29890005-vntzg\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.475316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.475244    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hzzm\" (UniqueName: \"kubernetes.io/projected/ce2ffc65-9dae-459d-8cae-e337d19522a2-kube-api-access-9hzzm\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.490279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.490253    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:00.496338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.496322    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:00.876912 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.876874    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"]
Oct 30 22:45:00.877288 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:45:00.877257    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2ffc65_9dae_459d_8cae_e337d19522a2.slice/crio-162e8b848edee00db7176a3e36425f4785351c398cb3fc9349c4719432f6aa92 WatchSource:0}: Error finding container 162e8b848edee00db7176a3e36425f4785351c398cb3fc9349c4719432f6aa92: Status 404 returned error can't find the container with id 162e8b848edee00db7176a3e36425f4785351c398cb3fc9349c4719432f6aa92
Oct 30 22:45:00.927040 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.927015    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"]
Oct 30 22:45:00.927590 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:45:00.927564    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e826a4_e256_43ad_8591_7812fa30010b.slice/crio-42dc843827f228915b8edeab8054ae1547b98634b657eb1d993e51fc1f420100 WatchSource:0}: Error finding container 42dc843827f228915b8edeab8054ae1547b98634b657eb1d993e51fc1f420100: Status 404 returned error can't find the container with id 42dc843827f228915b8edeab8054ae1547b98634b657eb1d993e51fc1f420100
Oct 30 22:45:00.955081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.955053    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" event={"ID":"a2e826a4-e256-43ad-8591-7812fa30010b","Type":"ContainerStarted","Data":"42dc843827f228915b8edeab8054ae1547b98634b657eb1d993e51fc1f420100"}
Oct 30 22:45:00.956340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:00.956320    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" event={"ID":"ce2ffc65-9dae-459d-8cae-e337d19522a2","Type":"ContainerStarted","Data":"162e8b848edee00db7176a3e36425f4785351c398cb3fc9349c4719432f6aa92"}
Oct 30 22:45:01.974750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:01.974664    5031 generic.go:356] "Generic (PLEG): container finished" podID="a2e826a4-e256-43ad-8591-7812fa30010b" containerID="372d73af77b6822da7e133d13d8c2d62e901bd1a8d7e7f941193d11562c95026" exitCode=60
Oct 30 22:45:01.975173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:01.974762    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" event={"ID":"a2e826a4-e256-43ad-8591-7812fa30010b","Type":"ContainerDied","Data":"372d73af77b6822da7e133d13d8c2d62e901bd1a8d7e7f941193d11562c95026"}
Oct 30 22:45:01.975173 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:01.975045    5031 scope.go:122] "RemoveContainer" containerID="372d73af77b6822da7e133d13d8c2d62e901bd1a8d7e7f941193d11562c95026"
Oct 30 22:45:01.977358 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:01.977329    5031 generic.go:356] "Generic (PLEG): container finished" podID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerID="a61e50ecc0ed361eb6774af66640ea72ad031f97d4b4c60cd54e9795992849e6" exitCode=60
Oct 30 22:45:01.977485 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:01.977421    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" event={"ID":"ce2ffc65-9dae-459d-8cae-e337d19522a2","Type":"ContainerDied","Data":"a61e50ecc0ed361eb6774af66640ea72ad031f97d4b4c60cd54e9795992849e6"}
Oct 30 22:45:01.977697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:01.977678    5031 scope.go:122] "RemoveContainer" containerID="a61e50ecc0ed361eb6774af66640ea72ad031f97d4b4c60cd54e9795992849e6"
Oct 30 22:45:02.811844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.811808    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:02.985858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.985823    5031 generic.go:356] "Generic (PLEG): container finished" podID="a2e826a4-e256-43ad-8591-7812fa30010b" containerID="0f7a1a160f7e4fd8af3a2ebbff6dc2550fc614d0fe692eb4bfe424a935d02592" exitCode=60
Oct 30 22:45:02.986333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.985876    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" event={"ID":"a2e826a4-e256-43ad-8591-7812fa30010b","Type":"ContainerDied","Data":"0f7a1a160f7e4fd8af3a2ebbff6dc2550fc614d0fe692eb4bfe424a935d02592"}
Oct 30 22:45:02.986333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.985974    5031 scope.go:122] "RemoveContainer" containerID="372d73af77b6822da7e133d13d8c2d62e901bd1a8d7e7f941193d11562c95026"
Oct 30 22:45:02.986333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.986316    5031 scope.go:122] "RemoveContainer" containerID="0f7a1a160f7e4fd8af3a2ebbff6dc2550fc614d0fe692eb4bfe424a935d02592"
Oct 30 22:45:02.986633 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:02.986607    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-vntzg_odh-ai-gateway-infra(a2e826a4-e256-43ad-8591-7812fa30010b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" podUID="a2e826a4-e256-43ad-8591-7812fa30010b"
Oct 30 22:45:02.988052 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.988030    5031 generic.go:356] "Generic (PLEG): container finished" podID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerID="b7c87cff93fe727e8320a5347bd19becac5b26487478cf4bfd6cbeb8cb921ce9" exitCode=60
Oct 30 22:45:02.988146 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.988109    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" event={"ID":"ce2ffc65-9dae-459d-8cae-e337d19522a2","Type":"ContainerDied","Data":"b7c87cff93fe727e8320a5347bd19becac5b26487478cf4bfd6cbeb8cb921ce9"}
Oct 30 22:45:02.988360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.988346    5031 scope.go:122] "RemoveContainer" containerID="b7c87cff93fe727e8320a5347bd19becac5b26487478cf4bfd6cbeb8cb921ce9"
Oct 30 22:45:02.988593 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:02.988576    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck_odh-ai-gateway-infra(ce2ffc65-9dae-459d-8cae-e337d19522a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2"
Oct 30 22:45:02.998381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:02.998359    5031 scope.go:122] "RemoveContainer" containerID="a61e50ecc0ed361eb6774af66640ea72ad031f97d4b4c60cd54e9795992849e6"
Oct 30 22:45:03.762169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:03.762131    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:45:03.996894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:03.996855    5031 scope.go:122] "RemoveContainer" containerID="0f7a1a160f7e4fd8af3a2ebbff6dc2550fc614d0fe692eb4bfe424a935d02592"
Oct 30 22:45:03.997396 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:03.997163    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-vntzg_odh-ai-gateway-infra(a2e826a4-e256-43ad-8591-7812fa30010b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" podUID="a2e826a4-e256-43ad-8591-7812fa30010b"
Oct 30 22:45:03.999081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:03.999058    5031 scope.go:122] "RemoveContainer" containerID="b7c87cff93fe727e8320a5347bd19becac5b26487478cf4bfd6cbeb8cb921ce9"
Oct 30 22:45:03.999398 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:03.999365    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck_odh-ai-gateway-infra(ce2ffc65-9dae-459d-8cae-e337d19522a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2"
Oct 30 22:45:07.855251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:07.855195    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:13.089029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:13.088992    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:14.645988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:14.645960    5031 scope.go:122] "RemoveContainer" containerID="b7c87cff93fe727e8320a5347bd19becac5b26487478cf4bfd6cbeb8cb921ce9"
Oct 30 22:45:15.073783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:15.073744    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" event={"ID":"ce2ffc65-9dae-459d-8cae-e337d19522a2","Type":"ContainerStarted","Data":"cb3f109d7cb034dfe4a1737eec8ed9986991638f37c446f20d3e3096bed17cf7"}
Oct 30 22:45:15.195524 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:15.195417    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" podStartSLOduration=15.1954024 podStartE2EDuration="15.1954024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:15.1938904 +0000 UTC m=+3326.718033164" watchObservedRunningTime="2026-10-30 22:45:15.1954024 +0000 UTC m=+3326.719545164"
Oct 30 22:45:16.081558 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:16.081526    5031 generic.go:356] "Generic (PLEG): container finished" podID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerID="cb3f109d7cb034dfe4a1737eec8ed9986991638f37c446f20d3e3096bed17cf7" exitCode=60
Oct 30 22:45:16.081968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:16.081599    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" event={"ID":"ce2ffc65-9dae-459d-8cae-e337d19522a2","Type":"ContainerDied","Data":"cb3f109d7cb034dfe4a1737eec8ed9986991638f37c446f20d3e3096bed17cf7"}
Oct 30 22:45:16.081968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:16.081658    5031 scope.go:122] "RemoveContainer" containerID="b7c87cff93fe727e8320a5347bd19becac5b26487478cf4bfd6cbeb8cb921ce9"
Oct 30 22:45:16.082070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:16.081972    5031 scope.go:122] "RemoveContainer" containerID="cb3f109d7cb034dfe4a1737eec8ed9986991638f37c446f20d3e3096bed17cf7"
Oct 30 22:45:16.082244 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:16.082200    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck_odh-ai-gateway-infra(ce2ffc65-9dae-459d-8cae-e337d19522a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2"
Oct 30 22:45:16.211174 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:16.211135    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"]
Oct 30 22:45:17.232724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.232697    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:17.289127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.289099    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2ffc65-9dae-459d-8cae-e337d19522a2-kube-api-access-9hzzm\" (UniqueName: \"kubernetes.io/projected/ce2ffc65-9dae-459d-8cae-e337d19522a2-kube-api-access-9hzzm\") pod \"ce2ffc65-9dae-459d-8cae-e337d19522a2\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") "
Oct 30 22:45:17.289365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.289230    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2ffc65-9dae-459d-8cae-e337d19522a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2ffc65-9dae-459d-8cae-e337d19522a2-maas-api-service-ca\") pod \"ce2ffc65-9dae-459d-8cae-e337d19522a2\" (UID: \"ce2ffc65-9dae-459d-8cae-e337d19522a2\") "
Oct 30 22:45:17.289571 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.289544    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2ffc65-9dae-459d-8cae-e337d19522a2-maas-api-service-ca" pod "ce2ffc65-9dae-459d-8cae-e337d19522a2" (UID: "ce2ffc65-9dae-459d-8cae-e337d19522a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:17.291554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.291526    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2ffc65-9dae-459d-8cae-e337d19522a2-kube-api-access-9hzzm" pod "ce2ffc65-9dae-459d-8cae-e337d19522a2" (UID: "ce2ffc65-9dae-459d-8cae-e337d19522a2"). InnerVolumeSpecName "kube-api-access-9hzzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.390665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.390636    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2ffc65-9dae-459d-8cae-e337d19522a2-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.390665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.390663    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hzzm\" (UniqueName: \"kubernetes.io/projected/ce2ffc65-9dae-459d-8cae-e337d19522a2-kube-api-access-9hzzm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.638337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:17.638303    5031 scope.go:122] "RemoveContainer" containerID="0f7a1a160f7e4fd8af3a2ebbff6dc2550fc614d0fe692eb4bfe424a935d02592"
Oct 30 22:45:18.098364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.098246    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" event={"ID":"a2e826a4-e256-43ad-8591-7812fa30010b","Type":"ContainerStarted","Data":"e21a2de38f195bb3ea6206af3b5cdff9764e2d2ead6ceb9eb7c695a4ac3a8750"}
Oct 30 22:45:18.100742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.100718    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"
Oct 30 22:45:18.100933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.100830    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck" event={"ID":"ce2ffc65-9dae-459d-8cae-e337d19522a2","Type":"ContainerDied","Data":"162e8b848edee00db7176a3e36425f4785351c398cb3fc9349c4719432f6aa92"}
Oct 30 22:45:18.100933 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.100894    5031 scope.go:122] "RemoveContainer" containerID="cb3f109d7cb034dfe4a1737eec8ed9986991638f37c446f20d3e3096bed17cf7"
Oct 30 22:45:18.121920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.121895    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:18.218289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.218187    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" podStartSLOduration=18.218174605 podStartE2EDuration="18.218174605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:18.217022183 +0000 UTC m=+3329.741164958" watchObservedRunningTime="2026-10-30 22:45:18.218174605 +0000 UTC m=+3329.742317368"
Oct 30 22:45:18.333600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.333570    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"]
Oct 30 22:45:18.340946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.340921    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ef21ee-2989000xggck"]
Oct 30 22:45:18.645140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.645099    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" path="/var/lib/kubelet/pods/ce2ffc65-9dae-459d-8cae-e337d19522a2/volumes"
Oct 30 22:45:18.661394 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661372    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"]
Oct 30 22:45:18.661840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661828    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.661887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661841    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.661887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661875    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.661887 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661880    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.661991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661890    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.661991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661895    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.661991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661969    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.661991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.661977    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:18.665340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.665327    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.710668 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.710648    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.710806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.710676    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.710806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.710694    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkzp8\" (UniqueName: \"kubernetes.io/projected/18cc876a-0246-4b93-8c3d-e385896958b7-kube-api-access-hkzp8\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.723528 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.722328    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:18.725541 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.725517    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"]
Oct 30 22:45:18.812488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.812405    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.812488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.812439    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.812488 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.812459    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkzp8\" (UniqueName: \"kubernetes.io/projected/18cc876a-0246-4b93-8c3d-e385896958b7-kube-api-access-hkzp8\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.813136 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.813115    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.814946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.814929    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.876449 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.876422    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkzp8\" (UniqueName: \"kubernetes.io/projected/18cc876a-0246-4b93-8c3d-e385896958b7-kube-api-access-hkzp8\") pod \"maas-api-revoke-sub-e81d21d9-rwz5j\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:18.978759 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:18.978728    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:19.108672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:19.108643    5031 generic.go:356] "Generic (PLEG): container finished" podID="a2e826a4-e256-43ad-8591-7812fa30010b" containerID="e21a2de38f195bb3ea6206af3b5cdff9764e2d2ead6ceb9eb7c695a4ac3a8750" exitCode=60
Oct 30 22:45:19.108856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:19.108711    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" event={"ID":"a2e826a4-e256-43ad-8591-7812fa30010b","Type":"ContainerDied","Data":"e21a2de38f195bb3ea6206af3b5cdff9764e2d2ead6ceb9eb7c695a4ac3a8750"}
Oct 30 22:45:19.108856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:19.108777    5031 scope.go:122] "RemoveContainer" containerID="0f7a1a160f7e4fd8af3a2ebbff6dc2550fc614d0fe692eb4bfe424a935d02592"
Oct 30 22:45:19.109185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:19.109163    5031 scope.go:122] "RemoveContainer" containerID="e21a2de38f195bb3ea6206af3b5cdff9764e2d2ead6ceb9eb7c695a4ac3a8750"
Oct 30 22:45:19.109499 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:19.109474    5031 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-vntzg_odh-ai-gateway-infra(a2e826a4-e256-43ad-8591-7812fa30010b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" podUID="a2e826a4-e256-43ad-8591-7812fa30010b"
Oct 30 22:45:19.229869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:19.229838    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"]
Oct 30 22:45:19.363031 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:19.363001    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"]
Oct 30 22:45:19.363765 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:45:19.363735    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cc876a_0246_4b93_8c3d_e385896958b7.slice/crio-e93667880a39ab3c9573e4188758f28a2fb704d80b9cfa65f31d2d723e67ab3f WatchSource:0}: Error finding container e93667880a39ab3c9573e4188758f28a2fb704d80b9cfa65f31d2d723e67ab3f: Status 404 returned error can't find the container with id e93667880a39ab3c9573e4188758f28a2fb704d80b9cfa65f31d2d723e67ab3f
Oct 30 22:45:20.119274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.119246    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j" event={"ID":"18cc876a-0246-4b93-8c3d-e385896958b7","Type":"ContainerStarted","Data":"75c0860fee63c7bf31b169aa0f8d50af2b0574b7119a8a01897211397ae6f1e3"}
Oct 30 22:45:20.119400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.119293    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j" event={"ID":"18cc876a-0246-4b93-8c3d-e385896958b7","Type":"ContainerStarted","Data":"e93667880a39ab3c9573e4188758f28a2fb704d80b9cfa65f31d2d723e67ab3f"}
Oct 30 22:45:20.274426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.274403    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:20.327915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.327887    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e826a4-e256-43ad-8591-7812fa30010b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e826a4-e256-43ad-8591-7812fa30010b-maas-api-service-ca\") pod \"a2e826a4-e256-43ad-8591-7812fa30010b\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") "
Oct 30 22:45:20.328078 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.327942    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e826a4-e256-43ad-8591-7812fa30010b-kube-api-access-ggw5q\" (UniqueName: \"kubernetes.io/projected/a2e826a4-e256-43ad-8591-7812fa30010b-kube-api-access-ggw5q\") pod \"a2e826a4-e256-43ad-8591-7812fa30010b\" (UID: \"a2e826a4-e256-43ad-8591-7812fa30010b\") "
Oct 30 22:45:20.328272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.328250    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e826a4-e256-43ad-8591-7812fa30010b-maas-api-service-ca" pod "a2e826a4-e256-43ad-8591-7812fa30010b" (UID: "a2e826a4-e256-43ad-8591-7812fa30010b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.330289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.330261    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e826a4-e256-43ad-8591-7812fa30010b-kube-api-access-ggw5q" pod "a2e826a4-e256-43ad-8591-7812fa30010b" (UID: "a2e826a4-e256-43ad-8591-7812fa30010b"). InnerVolumeSpecName "kube-api-access-ggw5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.358046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.358002    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j" podStartSLOduration=2.357989 podStartE2EDuration="2.357989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:20.356723223 +0000 UTC m=+3331.880865998" watchObservedRunningTime="2026-10-30 22:45:20.357989 +0000 UTC m=+3331.882131763"
Oct 30 22:45:20.429266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.429167    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e826a4-e256-43ad-8591-7812fa30010b-maas-api-service-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.429266 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:20.429191    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggw5q\" (UniqueName: \"kubernetes.io/projected/a2e826a4-e256-43ad-8591-7812fa30010b-kube-api-access-ggw5q\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.543096 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:20.543055    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:45:21.128263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:21.128197    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg" event={"ID":"a2e826a4-e256-43ad-8591-7812fa30010b","Type":"ContainerDied","Data":"42dc843827f228915b8edeab8054ae1547b98634b657eb1d993e51fc1f420100"}
Oct 30 22:45:21.128263 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:21.128236    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"
Oct 30 22:45:21.128542 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:21.128280    5031 scope.go:122] "RemoveContainer" containerID="e21a2de38f195bb3ea6206af3b5cdff9764e2d2ead6ceb9eb7c695a4ac3a8750"
Oct 30 22:45:21.130228 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:21.130185    5031 generic.go:356] "Generic (PLEG): container finished" podID="18cc876a-0246-4b93-8c3d-e385896958b7" containerID="75c0860fee63c7bf31b169aa0f8d50af2b0574b7119a8a01897211397ae6f1e3" exitCode=0
Oct 30 22:45:21.130343 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:21.130255    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j" event={"ID":"18cc876a-0246-4b93-8c3d-e385896958b7","Type":"ContainerDied","Data":"75c0860fee63c7bf31b169aa0f8d50af2b0574b7119a8a01897211397ae6f1e3"}
Oct 30 22:45:21.437348 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:21.437266    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"]
Oct 30 22:45:21.457341 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:21.457306    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-vntzg"]
Oct 30 22:45:22.291376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.291352    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:22.352484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.352446    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-openshift-ca\") pod \"18cc876a-0246-4b93-8c3d-e385896958b7\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") "
Oct 30 22:45:22.352484 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.352483    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cc876a-0246-4b93-8c3d-e385896958b7-kube-api-access-hkzp8\" (UniqueName: \"kubernetes.io/projected/18cc876a-0246-4b93-8c3d-e385896958b7-kube-api-access-hkzp8\") pod \"18cc876a-0246-4b93-8c3d-e385896958b7\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") "
Oct 30 22:45:22.352763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.352587    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-xks-ca\") pod \"18cc876a-0246-4b93-8c3d-e385896958b7\" (UID: \"18cc876a-0246-4b93-8c3d-e385896958b7\") "
Oct 30 22:45:22.352919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.352887    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-openshift-ca" pod "18cc876a-0246-4b93-8c3d-e385896958b7" (UID: "18cc876a-0246-4b93-8c3d-e385896958b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.353054 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.353029    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.354980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.354957    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cc876a-0246-4b93-8c3d-e385896958b7-kube-api-access-hkzp8" pod "18cc876a-0246-4b93-8c3d-e385896958b7" (UID: "18cc876a-0246-4b93-8c3d-e385896958b7"). InnerVolumeSpecName "kube-api-access-hkzp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.355074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.355015    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-xks-ca" pod "18cc876a-0246-4b93-8c3d-e385896958b7" (UID: "18cc876a-0246-4b93-8c3d-e385896958b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:22.454160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.454054    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkzp8\" (UniqueName: \"kubernetes.io/projected/18cc876a-0246-4b93-8c3d-e385896958b7-kube-api-access-hkzp8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.454160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.454091    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc876a-0246-4b93-8c3d-e385896958b7-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.644701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:22.644664    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" path="/var/lib/kubelet/pods/a2e826a4-e256-43ad-8591-7812fa30010b/volumes"
Oct 30 22:45:23.146345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:23.146310    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j" event={"ID":"18cc876a-0246-4b93-8c3d-e385896958b7","Type":"ContainerDied","Data":"e93667880a39ab3c9573e4188758f28a2fb704d80b9cfa65f31d2d723e67ab3f"}
Oct 30 22:45:23.146345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:23.146342    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"
Oct 30 22:45:23.146585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:23.146367    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e93667880a39ab3c9573e4188758f28a2fb704d80b9cfa65f31d2d723e67ab3f"
Oct 30 22:45:23.283465 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:23.283431    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:27.631946 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:27.631892    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:28.295512 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:28.295477    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:29.853790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:29.853761    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:30.340616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.340531    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"]
Oct 30 22:45:30.341023 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341010    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:45:30.341070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341024    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:45:30.341070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341037    5031 cpu_manager.go:414] "containerMap: removing container" podUID="18cc876a-0246-4b93-8c3d-e385896958b7" containerName="revoke-keys"
Oct 30 22:45:30.341070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341042    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cc876a-0246-4b93-8c3d-e385896958b7" containerName="revoke-keys"
Oct 30 22:45:30.341070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341054    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:45:30.341070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341059    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:45:30.341259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341130    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:45:30.341259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341140    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cc876a-0246-4b93-8c3d-e385896958b7" containerName="revoke-keys"
Oct 30 22:45:30.341259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341146    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:45:30.341259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341153    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:45:30.341259 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.341161    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2ffc65-9dae-459d-8cae-e337d19522a2" containerName="cleanup"
Oct 30 22:45:30.344719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.344703    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.403712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.403686    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"]
Oct 30 22:45:30.430953 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.430928    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.431109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.430997    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.431109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.431093    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5smc\" (UniqueName: \"kubernetes.io/projected/5768ded9-b1ed-4179-9d82-739ac213a9bf-kube-api-access-w5smc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.450486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.450451    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:30.450486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.450482    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:30.450630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.450499    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:45:30.450630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.450509    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:30.532693 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.532659    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.532853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.532746    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.532853 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.532778    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5smc\" (UniqueName: \"kubernetes.io/projected/5768ded9-b1ed-4179-9d82-739ac213a9bf-kube-api-access-w5smc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.533438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.533417    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.535256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.535234    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.596645 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.596613    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5smc\" (UniqueName: \"kubernetes.io/projected/5768ded9-b1ed-4179-9d82-739ac213a9bf-kube-api-access-w5smc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:30.657966 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:30.657923    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:31.040941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:31.040907    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"]
Oct 30 22:45:31.042438 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:45:31.042404    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5768ded9_b1ed_4179_9d82_739ac213a9bf.slice/crio-773fc164b8d8e627b9b267acc8088c172a2ef664edb45d7d4291a91dd8a41550 WatchSource:0}: Error finding container 773fc164b8d8e627b9b267acc8088c172a2ef664edb45d7d4291a91dd8a41550: Status 404 returned error can't find the container with id 773fc164b8d8e627b9b267acc8088c172a2ef664edb45d7d4291a91dd8a41550
Oct 30 22:45:31.198057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:31.198025    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb" event={"ID":"5768ded9-b1ed-4179-9d82-739ac213a9bf","Type":"ContainerStarted","Data":"773fc164b8d8e627b9b267acc8088c172a2ef664edb45d7d4291a91dd8a41550"}
Oct 30 22:45:31.754719 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:31.754685    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:45:32.206656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:32.206623    5031 generic.go:356] "Generic (PLEG): container finished" podID="5768ded9-b1ed-4179-9d82-739ac213a9bf" containerID="11d364244966c54c1c676f055347c4231ccfba7003e0ba5227a2b784abcc3852" exitCode=0
Oct 30 22:45:32.207065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:32.206703    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb" event={"ID":"5768ded9-b1ed-4179-9d82-739ac213a9bf","Type":"ContainerDied","Data":"11d364244966c54c1c676f055347c4231ccfba7003e0ba5227a2b784abcc3852"}
Oct 30 22:45:33.363507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.363482    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:33.462163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.462130    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-openshift-ca\") pod \"5768ded9-b1ed-4179-9d82-739ac213a9bf\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") "
Oct 30 22:45:33.462397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.462190    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5768ded9-b1ed-4179-9d82-739ac213a9bf-kube-api-access-w5smc\" (UniqueName: \"kubernetes.io/projected/5768ded9-b1ed-4179-9d82-739ac213a9bf-kube-api-access-w5smc\") pod \"5768ded9-b1ed-4179-9d82-739ac213a9bf\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") "
Oct 30 22:45:33.462397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.462308    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-xks-ca\") pod \"5768ded9-b1ed-4179-9d82-739ac213a9bf\" (UID: \"5768ded9-b1ed-4179-9d82-739ac213a9bf\") "
Oct 30 22:45:33.462604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.462580    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-openshift-ca" pod "5768ded9-b1ed-4179-9d82-739ac213a9bf" (UID: "5768ded9-b1ed-4179-9d82-739ac213a9bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:33.462683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.462665    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:33.464527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.464501    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-xks-ca" pod "5768ded9-b1ed-4179-9d82-739ac213a9bf" (UID: "5768ded9-b1ed-4179-9d82-739ac213a9bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:33.464647 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.464558    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5768ded9-b1ed-4179-9d82-739ac213a9bf-kube-api-access-w5smc" pod "5768ded9-b1ed-4179-9d82-739ac213a9bf" (UID: "5768ded9-b1ed-4179-9d82-739ac213a9bf"). InnerVolumeSpecName "kube-api-access-w5smc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:33.564272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.564202    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5smc\" (UniqueName: \"kubernetes.io/projected/5768ded9-b1ed-4179-9d82-739ac213a9bf-kube-api-access-w5smc\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:33.564272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.564266    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5768ded9-b1ed-4179-9d82-739ac213a9bf-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:33.735412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:33.735315    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:34.226202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:34.226173    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"
Oct 30 22:45:34.226397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:34.226196    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb" event={"ID":"5768ded9-b1ed-4179-9d82-739ac213a9bf","Type":"ContainerDied","Data":"773fc164b8d8e627b9b267acc8088c172a2ef664edb45d7d4291a91dd8a41550"}
Oct 30 22:45:34.226397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:34.226262    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="773fc164b8d8e627b9b267acc8088c172a2ef664edb45d7d4291a91dd8a41550"
Oct 30 22:45:34.920479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:34.920451    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:38.947008 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:38.946977    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:39.063098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:39.063061    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"]
Oct 30 22:45:39.113631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:39.113599    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88eb1481-qwnhq"]
Oct 30 22:45:40.644826 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:40.644791    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb8daa6-18c8-4c8b-a398-6e2ba634e32a" path="/var/lib/kubelet/pods/cbb8daa6-18c8-4c8b-a398-6e2ba634e32a/volumes"
Oct 30 22:45:40.854824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:40.854788    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"]
Oct 30 22:45:41.188230 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:41.188182    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:41.455238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:41.455137    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.435547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:43.435512    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:44.997139 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:44.997102    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:45.287411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:45.287319    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:45.553822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:45.553732    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"]
Oct 30 22:45:45.604352 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:45.604314    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef21ee-fckqb"]
Oct 30 22:45:45.855619 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:45.855580    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" podUID="044327ac-2d86-4265-9077-80353e48cbd4" containerName="payload-pre-processing" containerID="cri-o://81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84" gracePeriod=25
Oct 30 22:45:45.871747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:45.871717    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.109611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:46.109494    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.645518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:46.645485    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5768ded9-b1ed-4179-9d82-739ac213a9bf" path="/var/lib/kubelet/pods/5768ded9-b1ed-4179-9d82-739ac213a9bf/volumes"
Oct 30 22:45:48.263808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:48.263773    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:45:50.541098 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:50.541059    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:45:50.623678 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:50.623645    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" podUID="044327ac-2d86-4265-9077-80353e48cbd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:45:53.010837 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:53.010748    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:53.275889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:53.275797    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:53.304289 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:53.304263    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"]
Oct 30 22:45:53.304462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:53.304443    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy" containerID="cri-o://e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b" gracePeriod=30
Oct 30 22:45:53.536037 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:53.535958    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:58.503190 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.503162    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:58.580160 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.580096    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:45:58.621431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621400    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-podinfo\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.621604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621446    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-kube-api-access-sgsxx\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-kube-api-access-sgsxx\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.621604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621557    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-certs\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.621604 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621583    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-token\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-token\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.621748 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621642    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-envoy\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.621748 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621688    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-data\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.621860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621744    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-credential-socket\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.621860 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621843    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-certs" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:58.622102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622059    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-data" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:58.622102 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622082    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-credential-socket" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:58.622318 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.621783    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istiod-ca-cert\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.622387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622315    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-socket\") pod \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\" (UID: \"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5\") "
Oct 30 22:45:58.622453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622422    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istiod-ca-cert" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:58.622653 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622630    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-socket" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:58.622949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622928    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.622991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622958    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.622991 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622976    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.623068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.622991    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.623068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.623005    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.624056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.624022    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-podinfo" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:45:58.624177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.624153    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-envoy" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:58.624519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.624497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-kube-api-access-sgsxx" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "kube-api-access-sgsxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:58.624593 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.624526    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-token" pod "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" (UID: "24f7e8a7-16ff-47f0-9ab0-a28e60a76be5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:58.724319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.724281    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.724319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.724314    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsxx\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-kube-api-access-sgsxx\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.724319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.724326    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.724575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:58.724335    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:59.384506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.384474    5031 generic.go:356] "Generic (PLEG): container finished" podID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerID="e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b" exitCode=0
Oct 30 22:45:59.384713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.384536    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"
Oct 30 22:45:59.384713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.384554    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" event={"ID":"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5","Type":"ContainerDied","Data":"e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b"}
Oct 30 22:45:59.384713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.384632    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4" event={"ID":"24f7e8a7-16ff-47f0-9ab0-a28e60a76be5","Type":"ContainerDied","Data":"1ee8782e34d08ff0fddf6dbd40c5f2e35d280238ab92ea1364b4aceb98ba4bd3"}
Oct 30 22:45:59.384713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.384667    5031 scope.go:122] "RemoveContainer" containerID="e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b"
Oct 30 22:45:59.396856 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.396831    5031 scope.go:122] "RemoveContainer" containerID="75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118"
Oct 30 22:45:59.406450 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.406430    5031 scope.go:122] "RemoveContainer" containerID="e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b"
Oct 30 22:45:59.406717 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:59.406699    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b\": container with ID starting with e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b not found: ID does not exist" containerID="e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b"
Oct 30 22:45:59.406791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.406724    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b"} err="failed to get container status \"e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b\": rpc error: code = NotFound desc = could not find container \"e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b\": container with ID starting with e2102ccc767b51428d78005778a7b6b22f591c32d3c07b13cc85d14ff023ff9b not found: ID does not exist"
Oct 30 22:45:59.406791 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.406740    5031 scope.go:122] "RemoveContainer" containerID="75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118"
Oct 30 22:45:59.406966 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:45:59.406946    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118\": container with ID starting with 75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118 not found: ID does not exist" containerID="75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118"
Oct 30 22:45:59.407032 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.406976    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118"} err="failed to get container status \"75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118\": rpc error: code = NotFound desc = could not find container \"75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118\": container with ID starting with 75b0a9c9ee7db6e18c8c4441002c57f1ea5352a47a8396e0ca113cd8195c5118 not found: ID does not exist"
Oct 30 22:45:59.619877 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.619841    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"]
Oct 30 22:45:59.622633 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:45:59.622608    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4"]
Oct 30 22:46:00.623920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:00.623882    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" podUID="044327ac-2d86-4265-9077-80353e48cbd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:00.647473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:00.647444    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" path="/var/lib/kubelet/pods/24f7e8a7-16ff-47f0-9ab0-a28e60a76be5/volumes"
Oct 30 22:46:01.764220 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764171    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"]
Oct 30 22:46:01.764652 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764637    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5768ded9-b1ed-4179-9d82-739ac213a9bf" containerName="revoke-keys"
Oct 30 22:46:01.764702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764653    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5768ded9-b1ed-4179-9d82-739ac213a9bf" containerName="revoke-keys"
Oct 30 22:46:01.764702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764672    5031 cpu_manager.go:414] "containerMap: removing container" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy"
Oct 30 22:46:01.764702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764677    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy"
Oct 30 22:46:01.764702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764692    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:46:01.764702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764698    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e826a4-e256-43ad-8591-7812fa30010b" containerName="cleanup"
Oct 30 22:46:01.764859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764707    5031 cpu_manager.go:414] "containerMap: removing container" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy"
Oct 30 22:46:01.764859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764712    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy"
Oct 30 22:46:01.764859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764790    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5768ded9-b1ed-4179-9d82-739ac213a9bf" containerName="revoke-keys"
Oct 30 22:46:01.764859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764798    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy"
Oct 30 22:46:01.764859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.764806    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f7e8a7-16ff-47f0-9ab0-a28e60a76be5" containerName="istio-proxy"
Oct 30 22:46:01.769540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.769522    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.857955 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.857920    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-home\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.858105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.857985    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tls-certs\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.858105 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.858085    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tmp-dir\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.858245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.858158    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-dshm\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.858245 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.858183    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfcln\" (UniqueName: \"kubernetes.io/projected/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-kube-api-access-gfcln\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.858337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.858269    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-model-cache\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.875399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.875374    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e06a579c-kserve-self-signed-certs\""
Oct 30 22:46:01.875951 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.875927    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"]
Oct 30 22:46:01.929591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.929560    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:01.959106 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959079    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tls-certs\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959114    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tmp-dir\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-dshm\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959177    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfcln\" (UniqueName: \"kubernetes.io/projected/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-kube-api-access-gfcln\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959238    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-model-cache\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959269    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-home\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959724 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959701    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-home\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959709    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tmp-dir\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.959806 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.959755    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-model-cache\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.961754 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.961731    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-dshm\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:01.962030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:01.962012    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tls-certs\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:02.023914 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:02.023850    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfcln\" (UniqueName: \"kubernetes.io/projected/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-kube-api-access-gfcln\") pod \"e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") " pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:02.081685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:02.081653    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:02.474776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:02.474748    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"]
Oct 30 22:46:02.476105 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:46:02.476080    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34417c7c_b6a1_4512_a5c9_02b92b7e7e85.slice/crio-499ae869d514e8518be370bebe31fddf54d9c1706e4f50a3227f956fb7440afa WatchSource:0}: Error finding container 499ae869d514e8518be370bebe31fddf54d9c1706e4f50a3227f956fb7440afa: Status 404 returned error can't find the container with id 499ae869d514e8518be370bebe31fddf54d9c1706e4f50a3227f956fb7440afa
Oct 30 22:46:03.413776 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:03.413738    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk" event={"ID":"34417c7c-b6a1-4512-a5c9-02b92b7e7e85","Type":"ContainerStarted","Data":"4d0958cfeeb8bbad7faa7235b782396b99c4269532f2b26a3d97992756ce5518"}
Oct 30 22:46:03.414233 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:03.413785    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk" event={"ID":"34417c7c-b6a1-4512-a5c9-02b92b7e7e85","Type":"ContainerStarted","Data":"499ae869d514e8518be370bebe31fddf54d9c1706e4f50a3227f956fb7440afa"}
Oct 30 22:46:03.536473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:03.536409    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk" podStartSLOduration=2.536391509 podStartE2EDuration="2.536391509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:03.534374626 +0000 UTC m=+3375.058517390" watchObservedRunningTime="2026-10-30 22:46:03.536391509 +0000 UTC m=+3375.060534274"
Oct 30 22:46:04.081882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:04.081848    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:06.129185 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.129117    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:46:06.201844 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.201815    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044327ac-2d86-4265-9077-80353e48cbd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044327ac-2d86-4265-9077-80353e48cbd4-plugins-config-volume\") pod \"044327ac-2d86-4265-9077-80353e48cbd4\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") "
Oct 30 22:46:06.202012 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.201947    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044327ac-2d86-4265-9077-80353e48cbd4-kube-api-access-48cdb\" (UniqueName: \"kubernetes.io/projected/044327ac-2d86-4265-9077-80353e48cbd4-kube-api-access-48cdb\") pod \"044327ac-2d86-4265-9077-80353e48cbd4\" (UID: \"044327ac-2d86-4265-9077-80353e48cbd4\") "
Oct 30 22:46:06.202187 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.202165    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044327ac-2d86-4265-9077-80353e48cbd4-plugins-config-volume" pod "044327ac-2d86-4265-9077-80353e48cbd4" (UID: "044327ac-2d86-4265-9077-80353e48cbd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:06.204238 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.204187    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044327ac-2d86-4265-9077-80353e48cbd4-kube-api-access-48cdb" pod "044327ac-2d86-4265-9077-80353e48cbd4" (UID: "044327ac-2d86-4265-9077-80353e48cbd4"). InnerVolumeSpecName "kube-api-access-48cdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:06.303406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.303375    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48cdb\" (UniqueName: \"kubernetes.io/projected/044327ac-2d86-4265-9077-80353e48cbd4-kube-api-access-48cdb\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:46:06.303406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.303400    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044327ac-2d86-4265-9077-80353e48cbd4-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:46:06.436061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.435962    5031 generic.go:356] "Generic (PLEG): container finished" podID="044327ac-2d86-4265-9077-80353e48cbd4" containerID="81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84" exitCode=0
Oct 30 22:46:06.436061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.436009    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" event={"ID":"044327ac-2d86-4265-9077-80353e48cbd4","Type":"ContainerDied","Data":"81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84"}
Oct 30 22:46:06.436061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.436013    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"
Oct 30 22:46:06.436061 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.436062    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq" event={"ID":"044327ac-2d86-4265-9077-80353e48cbd4","Type":"ContainerDied","Data":"8ce556de3c8dd607d4a5672e4aa35c6d55d5124c84f975914a0b423487c16f48"}
Oct 30 22:46:06.436452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.436091    5031 scope.go:122] "RemoveContainer" containerID="81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84"
Oct 30 22:46:06.448940 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.448913    5031 scope.go:122] "RemoveContainer" containerID="81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84"
Oct 30 22:46:06.449275 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:46:06.449253    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84\": container with ID starting with 81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84 not found: ID does not exist" containerID="81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84"
Oct 30 22:46:06.449371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.449286    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84"} err="failed to get container status \"81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84\": rpc error: code = NotFound desc = could not find container \"81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84\": container with ID starting with 81bab3b112548714f96cad13b788562c6268a44774ebb1aa3143fda60d5d3a84 not found: ID does not exist"
Oct 30 22:46:06.556984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.556948    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:46:06.606992 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.606953    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"]
Oct 30 22:46:06.609868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.609848    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq"]
Oct 30 22:46:06.646073 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:06.646044    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044327ac-2d86-4265-9077-80353e48cbd4" path="/var/lib/kubelet/pods/044327ac-2d86-4265-9077-80353e48cbd4/volumes"
Oct 30 22:46:12.082148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:12.082108    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:12.094861 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:12.094837    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:12.579399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:12.579371    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:46:12.745416 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:12.745381    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:13.364090 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:13.364060    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:17.057134 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:17.057090    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"]
Oct 30 22:46:17.108972 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:17.108937    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lt4nr"]
Oct 30 22:46:18.644479 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:18.644445    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184039f7-945f-4c45-9ec2-9c67356a59ff" path="/var/lib/kubelet/pods/184039f7-945f-4c45-9ec2-9c67356a59ff/volumes"
Oct 30 22:46:20.539863 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:46:20.539817    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:46:29.800483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:29.800449    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:45.266177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.266139    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"]
Oct 30 22:46:45.266681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.266644    5031 cpu_manager.go:414] "containerMap: removing container" podUID="044327ac-2d86-4265-9077-80353e48cbd4" containerName="payload-pre-processing"
Oct 30 22:46:45.266681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.266659    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044327ac-2d86-4265-9077-80353e48cbd4" containerName="payload-pre-processing"
Oct 30 22:46:45.266801 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.266756    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="044327ac-2d86-4265-9077-80353e48cbd4" containerName="payload-pre-processing"
Oct 30 22:46:45.271255 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.271229    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.310458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.310411    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-dshm\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.310865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.310668    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-model-cache\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.310865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.310747    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tls-certs\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.310865 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.310843    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-kube-api-access-wqb5f\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.311019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.310880    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tmp-dir\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.311068 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.311028    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-home\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.377727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.377695    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e06a579c-kserve-self-signed-certs\""
Oct 30 22:46:45.380502 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.380474    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"]
Oct 30 22:46:45.412065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.412027    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-dshm\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.412065 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.412077    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-model-cache\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.412398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.412110    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tls-certs\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.412398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.412157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-kube-api-access-wqb5f\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.412398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.412182    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tmp-dir\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.412398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.412341    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-home\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.412707 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.412673    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tmp-dir\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.413129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.413104    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-home\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.413284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.413269    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-model-cache\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.420070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.415471    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tls-certs\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.420070 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.415708    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-dshm\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.468763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.468707    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:45.476118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.476088    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-kube-api-access-wqb5f\") pod \"e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") " pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.584057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.583968    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:45.690157 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.690124    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:45.970473 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:45.970445    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"]
Oct 30 22:46:45.972168 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:46:45.972146    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba8a6be7_cea7_496a_94bd_a2e79c8346d7.slice/crio-b1699196a9f51ec1ca01f2af20fa8d386f99366fd2d3e8ef90cc5132b9328301 WatchSource:0}: Error finding container b1699196a9f51ec1ca01f2af20fa8d386f99366fd2d3e8ef90cc5132b9328301: Status 404 returned error can't find the container with id b1699196a9f51ec1ca01f2af20fa8d386f99366fd2d3e8ef90cc5132b9328301
Oct 30 22:46:46.767265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:46.767200    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7" event={"ID":"ba8a6be7-cea7-496a-94bd-a2e79c8346d7","Type":"ContainerStarted","Data":"b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00"}
Oct 30 22:46:46.767798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:46.767275    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7" event={"ID":"ba8a6be7-cea7-496a-94bd-a2e79c8346d7","Type":"ContainerStarted","Data":"b1699196a9f51ec1ca01f2af20fa8d386f99366fd2d3e8ef90cc5132b9328301"}
Oct 30 22:46:46.891582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:46.891530    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7" podStartSLOduration=1.8915162269999999 podStartE2EDuration="1.891516227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:46.89053442 +0000 UTC m=+3418.414677184" watchObservedRunningTime="2026-10-30 22:46:46.891516227 +0000 UTC m=+3418.415658987"
Oct 30 22:46:47.584978 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:47.584945    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:50.156988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:50.156952    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:46:50.544880 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:46:50.544766    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:46:52.759795 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:52.759756    5031 scope.go:122] "RemoveContainer" containerID="1c99681ecbbddfead5e2d5d3ca7e397171e6e9217b0c166abdcd4e0377726f5e"
Oct 30 22:46:52.795407 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:52.795386    5031 scope.go:122] "RemoveContainer" containerID="d62f03aa4cf47b19cf6e3d8045a136fcfd326c223b56a3f99e4c751bde8fbf0f"
Oct 30 22:46:55.585071 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:55.585041    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:55.597446 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:55.597421    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:55.915687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:55.915656    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:46:58.799041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:46:58.799002    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:18.575641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:18.575600    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:47:20.602079 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:47:20.602041    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:47:21.878585 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:21.878543    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:47:21.879304 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:21.879268    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:47:29.841710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:29.841633    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"]
Oct 30 22:47:29.934503 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:29.934470    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:34.285395 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:34.285359    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:47:36.878519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:36.878462    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-t54jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:47:36.879188 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:36.879140    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj" podUID="cc3fa4fe-d521-4a83-b82d-27bf5dda0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:47:38.624546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:38.624516    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"]
Oct 30 22:47:38.742249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:38.740964    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:44.872234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:44.871372    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk" podUID="34417c7c-b6a1-4512-a5c9-02b92b7e7e85" containerName="main" containerID="cri-o://4d0958cfeeb8bbad7faa7235b782396b99c4269532f2b26a3d97992756ce5518" gracePeriod=45
Oct 30 22:47:45.302406 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.302363    5031 generic.go:356] "Generic (PLEG): container finished" podID="34417c7c-b6a1-4512-a5c9-02b92b7e7e85" containerID="4d0958cfeeb8bbad7faa7235b782396b99c4269532f2b26a3d97992756ce5518" exitCode=0
Oct 30 22:47:45.302871 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.302849    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk" event={"ID":"34417c7c-b6a1-4512-a5c9-02b92b7e7e85","Type":"ContainerDied","Data":"4d0958cfeeb8bbad7faa7235b782396b99c4269532f2b26a3d97992756ce5518"}
Oct 30 22:47:45.379301 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.378117    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:47:45.395993 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.395956    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-dshm\") pod \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") "
Oct 30 22:47:45.396169 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.396047    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tls-certs\" (UniqueName: \"kubernetes.io/secret/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tls-certs\") pod \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") "
Oct 30 22:47:45.396753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.396715    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-kube-api-access-gfcln\" (UniqueName: \"kubernetes.io/projected/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-kube-api-access-gfcln\") pod \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") "
Oct 30 22:47:45.397089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397068    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-home\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-home\") pod \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") "
Oct 30 22:47:45.397234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397150    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-model-cache\") pod \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") "
Oct 30 22:47:45.397294 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397258    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tmp-dir\") pod \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\" (UID: \"34417c7c-b6a1-4512-a5c9-02b92b7e7e85\") "
Oct 30 22:47:45.397671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397490    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-model-cache" pod "34417c7c-b6a1-4512-a5c9-02b92b7e7e85" (UID: "34417c7c-b6a1-4512-a5c9-02b92b7e7e85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:45.397671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397546    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-home" pod "34417c7c-b6a1-4512-a5c9-02b92b7e7e85" (UID: "34417c7c-b6a1-4512-a5c9-02b92b7e7e85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:45.397671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397640    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tmp-dir" pod "34417c7c-b6a1-4512-a5c9-02b92b7e7e85" (UID: "34417c7c-b6a1-4512-a5c9-02b92b7e7e85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:45.397868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397819    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.397868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397840    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.397868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.397854    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.399219 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.399178    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tls-certs" pod "34417c7c-b6a1-4512-a5c9-02b92b7e7e85" (UID: "34417c7c-b6a1-4512-a5c9-02b92b7e7e85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:45.399455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.399431    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-dshm" pod "34417c7c-b6a1-4512-a5c9-02b92b7e7e85" (UID: "34417c7c-b6a1-4512-a5c9-02b92b7e7e85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:45.400028 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.400009    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-kube-api-access-gfcln" pod "34417c7c-b6a1-4512-a5c9-02b92b7e7e85" (UID: "34417c7c-b6a1-4512-a5c9-02b92b7e7e85"). InnerVolumeSpecName "kube-api-access-gfcln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:45.504461 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.504421    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.504461 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.504457    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.504723 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:45.504474    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfcln\" (UniqueName: \"kubernetes.io/projected/34417c7c-b6a1-4512-a5c9-02b92b7e7e85-kube-api-access-gfcln\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.310709 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:46.310677    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"
Oct 30 22:47:46.311176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:46.310721    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk" event={"ID":"34417c7c-b6a1-4512-a5c9-02b92b7e7e85","Type":"ContainerDied","Data":"499ae869d514e8518be370bebe31fddf54d9c1706e4f50a3227f956fb7440afa"}
Oct 30 22:47:46.311176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:46.310774    5031 scope.go:122] "RemoveContainer" containerID="4d0958cfeeb8bbad7faa7235b782396b99c4269532f2b26a3d97992756ce5518"
Oct 30 22:47:46.481316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:46.481283    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"]
Oct 30 22:47:46.497455 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:46.497431    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e06a579c-kserve-68cd4cb549-z2gjk"]
Oct 30 22:47:46.644374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:46.644344    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34417c7c-b6a1-4512-a5c9-02b92b7e7e85" path="/var/lib/kubelet/pods/34417c7c-b6a1-4512-a5c9-02b92b7e7e85/volumes"
Oct 30 22:47:50.068366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:50.068328    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:47:50.541079 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:47:50.541045    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:47:52.951389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:52.951359    5031 scope.go:122] "RemoveContainer" containerID="302e2566f7194e54e03d546272fc7ee715cdcdb2fa4b44de1277da64786ef004"
Oct 30 22:47:53.645840 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.645806    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7" podUID="ba8a6be7-cea7-496a-94bd-a2e79c8346d7" containerName="main" containerID="cri-o://b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00" gracePeriod=45
Oct 30 22:47:53.912717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.912660    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:47:53.991055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991027    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tmp-dir\") pod \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") "
Oct 30 22:47:53.991545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991122    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-kube-api-access-wqb5f\") pod \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") "
Oct 30 22:47:53.991545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991166    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tls-certs\") pod \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") "
Oct 30 22:47:53.991545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991199    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-dshm\") pod \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") "
Oct 30 22:47:53.991545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991268    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-model-cache\") pod \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") "
Oct 30 22:47:53.991545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991300    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-home\") pod \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\" (UID: \"ba8a6be7-cea7-496a-94bd-a2e79c8346d7\") "
Oct 30 22:47:53.991545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991325    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tmp-dir" pod "ba8a6be7-cea7-496a-94bd-a2e79c8346d7" (UID: "ba8a6be7-cea7-496a-94bd-a2e79c8346d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.991545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991528    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-model-cache" pod "ba8a6be7-cea7-496a-94bd-a2e79c8346d7" (UID: "ba8a6be7-cea7-496a-94bd-a2e79c8346d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.992141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991557    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-home" pod "ba8a6be7-cea7-496a-94bd-a2e79c8346d7" (UID: "ba8a6be7-cea7-496a-94bd-a2e79c8346d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.992141 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.991584    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tmp-dir\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:53.993559 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.993533    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-dshm" pod "ba8a6be7-cea7-496a-94bd-a2e79c8346d7" (UID: "ba8a6be7-cea7-496a-94bd-a2e79c8346d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.993683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.993613    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-kube-api-access-wqb5f" pod "ba8a6be7-cea7-496a-94bd-a2e79c8346d7" (UID: "ba8a6be7-cea7-496a-94bd-a2e79c8346d7"). InnerVolumeSpecName "kube-api-access-wqb5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:53.993683 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:53.993626    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tls-certs" pod "ba8a6be7-cea7-496a-94bd-a2e79c8346d7" (UID: "ba8a6be7-cea7-496a-94bd-a2e79c8346d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:54.092362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.092330    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-home\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.092362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.092358    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqb5f\" (UniqueName: \"kubernetes.io/projected/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-kube-api-access-wqb5f\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.092362 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.092368    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-tls-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.092586 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.092378    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-dshm\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.092586 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.092386    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a6be7-cea7-496a-94bd-a2e79c8346d7-model-cache\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.362950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.362917    5031 generic.go:356] "Generic (PLEG): container finished" podID="ba8a6be7-cea7-496a-94bd-a2e79c8346d7" containerID="b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00" exitCode=0
Oct 30 22:47:54.363128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.362973    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"
Oct 30 22:47:54.363128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.362992    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7" event={"ID":"ba8a6be7-cea7-496a-94bd-a2e79c8346d7","Type":"ContainerDied","Data":"b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00"}
Oct 30 22:47:54.363128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.363039    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7" event={"ID":"ba8a6be7-cea7-496a-94bd-a2e79c8346d7","Type":"ContainerDied","Data":"b1699196a9f51ec1ca01f2af20fa8d386f99366fd2d3e8ef90cc5132b9328301"}
Oct 30 22:47:54.363128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.363056    5031 scope.go:122] "RemoveContainer" containerID="b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00"
Oct 30 22:47:54.373995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.373978    5031 scope.go:122] "RemoveContainer" containerID="b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00"
Oct 30 22:47:54.374281 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:47:54.374260    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00\": container with ID starting with b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00 not found: ID does not exist" containerID="b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00"
Oct 30 22:47:54.374349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.374290    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00"} err="failed to get container status \"b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00\": rpc error: code = NotFound desc = could not find container \"b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00\": container with ID starting with b7db4dd0b4b9af1e51caf65713adb62c0a9e6b685010cffa5d28580ccf45cb00 not found: ID does not exist"
Oct 30 22:47:54.535078 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.535044    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"]
Oct 30 22:47:54.541131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.541107    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7"]
Oct 30 22:47:54.645366 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.645295    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8a6be7-cea7-496a-94bd-a2e79c8346d7" path="/var/lib/kubelet/pods/ba8a6be7-cea7-496a-94bd-a2e79c8346d7/volumes"
Oct 30 22:47:54.902968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:54.902877    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-e06a579c-kserve-d4789b5cc-d6ds7" podUID="ba8a6be7-cea7-496a-94bd-a2e79c8346d7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:47:59.247099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:59.247067    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:59.507834 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:47:59.507731    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:02.163637 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:02.163599    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:48:12.455862 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.455826    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"]
Oct 30 22:48:12.456400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.456378    5031 cpu_manager.go:414] "containerMap: removing container" podUID="34417c7c-b6a1-4512-a5c9-02b92b7e7e85" containerName="main"
Oct 30 22:48:12.456400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.456395    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34417c7c-b6a1-4512-a5c9-02b92b7e7e85" containerName="main"
Oct 30 22:48:12.456499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.456423    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ba8a6be7-cea7-496a-94bd-a2e79c8346d7" containerName="main"
Oct 30 22:48:12.456499 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.456429    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8a6be7-cea7-496a-94bd-a2e79c8346d7" containerName="main"
Oct 30 22:48:12.456582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.456534    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="34417c7c-b6a1-4512-a5c9-02b92b7e7e85" containerName="main"
Oct 30 22:48:12.456582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.456548    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8a6be7-cea7-496a-94bd-a2e79c8346d7" containerName="main"
Oct 30 22:48:12.461368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.461351    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.519438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.519406    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"]
Oct 30 22:48:12.566159 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.566130    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:12.566354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.566172    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:12.566354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.566184    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:12.566354 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.566203    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzs6w\""
Oct 30 22:48:12.568797 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.568767    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.568930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.568809    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.568930 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.568833    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9mf\" (UniqueName: \"kubernetes.io/projected/1a69a495-3a91-46ed-abe8-38b96966cd8b-kube-api-access-fq9mf\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.669656 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.669623    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.669848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.669669    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.669848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.669692    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9mf\" (UniqueName: \"kubernetes.io/projected/1a69a495-3a91-46ed-abe8-38b96966cd8b-kube-api-access-fq9mf\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.670390 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.670365    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.672308 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.672288    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.734691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.734612    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9mf\" (UniqueName: \"kubernetes.io/projected/1a69a495-3a91-46ed-abe8-38b96966cd8b-kube-api-access-fq9mf\") pod \"maas-api-revoke-sub-ff56c627-6fdqs\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:12.776438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:12.776404    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:13.154976 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:13.154948    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"]
Oct 30 22:48:13.156521 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:48:13.156497    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a69a495_3a91_46ed_abe8_38b96966cd8b.slice/crio-a1e5f5e54591a62c7a16f1cf04835abbcaa422a2ab20deab292b1e0a5e20dbb9 WatchSource:0}: Error finding container a1e5f5e54591a62c7a16f1cf04835abbcaa422a2ab20deab292b1e0a5e20dbb9: Status 404 returned error can't find the container with id a1e5f5e54591a62c7a16f1cf04835abbcaa422a2ab20deab292b1e0a5e20dbb9
Oct 30 22:48:13.531968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:13.531930    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs" event={"ID":"1a69a495-3a91-46ed-abe8-38b96966cd8b","Type":"ContainerStarted","Data":"524b04a941804d67a8e438572d1f7899949b136612b8dea405f5dd0f4dd49658"}
Oct 30 22:48:13.532403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:13.531988    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs" event={"ID":"1a69a495-3a91-46ed-abe8-38b96966cd8b","Type":"ContainerStarted","Data":"a1e5f5e54591a62c7a16f1cf04835abbcaa422a2ab20deab292b1e0a5e20dbb9"}
Oct 30 22:48:13.653055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:13.653006    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs" podStartSLOduration=1.652992666 podStartE2EDuration="1.652992666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:13.652271893 +0000 UTC m=+3505.176414676" watchObservedRunningTime="2026-10-30 22:48:13.652992666 +0000 UTC m=+3505.177135441"
Oct 30 22:48:14.539613 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:14.539580    5031 generic.go:356] "Generic (PLEG): container finished" podID="1a69a495-3a91-46ed-abe8-38b96966cd8b" containerID="524b04a941804d67a8e438572d1f7899949b136612b8dea405f5dd0f4dd49658" exitCode=0
Oct 30 22:48:14.540027 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:14.539683    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs" event={"ID":"1a69a495-3a91-46ed-abe8-38b96966cd8b","Type":"ContainerDied","Data":"524b04a941804d67a8e438572d1f7899949b136612b8dea405f5dd0f4dd49658"}
Oct 30 22:48:15.689695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.689666    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:15.801178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.801121    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a69a495-3a91-46ed-abe8-38b96966cd8b-kube-api-access-fq9mf\" (UniqueName: \"kubernetes.io/projected/1a69a495-3a91-46ed-abe8-38b96966cd8b-kube-api-access-fq9mf\") pod \"1a69a495-3a91-46ed-abe8-38b96966cd8b\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") "
Oct 30 22:48:15.801178 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.801153    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-xks-ca\") pod \"1a69a495-3a91-46ed-abe8-38b96966cd8b\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") "
Oct 30 22:48:15.801388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.801262    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-openshift-ca\") pod \"1a69a495-3a91-46ed-abe8-38b96966cd8b\" (UID: \"1a69a495-3a91-46ed-abe8-38b96966cd8b\") "
Oct 30 22:48:15.801617 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.801591    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-openshift-ca" pod "1a69a495-3a91-46ed-abe8-38b96966cd8b" (UID: "1a69a495-3a91-46ed-abe8-38b96966cd8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:15.803421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.803391    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-xks-ca" pod "1a69a495-3a91-46ed-abe8-38b96966cd8b" (UID: "1a69a495-3a91-46ed-abe8-38b96966cd8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:15.803506 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.803488    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a69a495-3a91-46ed-abe8-38b96966cd8b-kube-api-access-fq9mf" pod "1a69a495-3a91-46ed-abe8-38b96966cd8b" (UID: "1a69a495-3a91-46ed-abe8-38b96966cd8b"). InnerVolumeSpecName "kube-api-access-fq9mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:15.902742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.902711    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq9mf\" (UniqueName: \"kubernetes.io/projected/1a69a495-3a91-46ed-abe8-38b96966cd8b-kube-api-access-fq9mf\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:15.902742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.902737    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:15.902929 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:15.902746    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a69a495-3a91-46ed-abe8-38b96966cd8b-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:16.552792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:16.552757    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs" event={"ID":"1a69a495-3a91-46ed-abe8-38b96966cd8b","Type":"ContainerDied","Data":"a1e5f5e54591a62c7a16f1cf04835abbcaa422a2ab20deab292b1e0a5e20dbb9"}
Oct 30 22:48:16.552980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:16.552838    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e5f5e54591a62c7a16f1cf04835abbcaa422a2ab20deab292b1e0a5e20dbb9"
Oct 30 22:48:16.552980 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:16.552770    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"
Oct 30 22:48:20.541191 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:48:20.541157    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:48:22.069793 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:22.069756    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"]
Oct 30 22:48:22.120267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:22.120239    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cv7x"]
Oct 30 22:48:22.644334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:22.644301    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523a7889-59a7-4c2a-a153-6e6afe28a933" path="/var/lib/kubelet/pods/523a7889-59a7-4c2a-a153-6e6afe28a933/volumes"
Oct 30 22:48:25.368179 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:25.368144    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:48:32.549954 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:32.549909    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:33.020997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.020965    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"]
Oct 30 22:48:33.021444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.021431    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1a69a495-3a91-46ed-abe8-38b96966cd8b" containerName="revoke-keys"
Oct 30 22:48:33.021557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.021446    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a69a495-3a91-46ed-abe8-38b96966cd8b" containerName="revoke-keys"
Oct 30 22:48:33.021557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.021535    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a69a495-3a91-46ed-abe8-38b96966cd8b" containerName="revoke-keys"
Oct 30 22:48:33.024949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.024935    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.080863 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.080824    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7e1fe5c2-openshift-default-dockercfg-79pq5\""
Oct 30 22:48:33.133799 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.133763    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"]
Oct 30 22:48:33.183115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183076    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-istio-token\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183122    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-socket\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183154    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-certs\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183220    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-credential-socket\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183284    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-envoy\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183409    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fee8d26-0788-4757-b949-acc1ae153f55-istio-podinfo\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183449    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfdr\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-kube-api-access-bdfdr\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183562    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fee8d26-0788-4757-b949-acc1ae153f55-istiod-ca-cert\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.183752 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.183619    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-data\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.188101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.188076    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:33.284941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.284836    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-istio-token\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.284941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.284904    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-socket\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.284941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.284921    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-certs\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.284941 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.284936    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-credential-socket\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.284960    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-envoy\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285018    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fee8d26-0788-4757-b949-acc1ae153f55-istio-podinfo\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285038    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfdr\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-kube-api-access-bdfdr\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285099    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fee8d26-0788-4757-b949-acc1ae153f55-istiod-ca-cert\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285346 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285136    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-data\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285436    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-certs\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285481    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-socket\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285597    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-data\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285737 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285719    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fee8d26-0788-4757-b949-acc1ae153f55-istiod-ca-cert\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.285950 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.285826    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-credential-socket\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.287576 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.287550    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-envoy\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.287931 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.287909    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fee8d26-0788-4757-b949-acc1ae153f55-istio-podinfo\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.348923 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.348872    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-istio-token\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.349087 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.349020    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfdr\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-kube-api-access-bdfdr\") pod \"e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") " pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:33.461257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.461195    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:33.638121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:33.638079    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:34.074588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:34.074554    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"]
Oct 30 22:48:34.076438 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:48:34.076413    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fee8d26_0788_4757_b949_acc1ae153f55.slice/crio-641a6fc2f9fcc2891ce77545a891d5cfe33e54e24bebae56b5f968aa4ded2a5a WatchSource:0}: Error finding container 641a6fc2f9fcc2891ce77545a891d5cfe33e54e24bebae56b5f968aa4ded2a5a: Status 404 returned error can't find the container with id 641a6fc2f9fcc2891ce77545a891d5cfe33e54e24bebae56b5f968aa4ded2a5a
Oct 30 22:48:34.078535 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:34.078504    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:34.078596 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:34.078568    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:34.078639 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:34.078599    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:34.658089 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:34.658054    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" event={"ID":"7fee8d26-0788-4757-b949-acc1ae153f55","Type":"ContainerStarted","Data":"a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c"}
Oct 30 22:48:34.658511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:34.658111    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" event={"ID":"7fee8d26-0788-4757-b949-acc1ae153f55","Type":"ContainerStarted","Data":"641a6fc2f9fcc2891ce77545a891d5cfe33e54e24bebae56b5f968aa4ded2a5a"}
Oct 30 22:48:34.836567 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:34.836512    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" podStartSLOduration=2.836497557 podStartE2EDuration="2.836497557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:34.833510078 +0000 UTC m=+3526.357652858" watchObservedRunningTime="2026-10-30 22:48:34.836497557 +0000 UTC m=+3526.360640320"
Oct 30 22:48:35.638434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:35.638400    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:35.643262 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:35.643240    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:35.663621 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:35.663593    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:35.664557 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:35.664536    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:48:37.086428 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:37.086345    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:37.353609 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:37.353577    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:40.438356 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.438322    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:40.669762 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.669722    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:40.774883 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.774852    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"]
Oct 30 22:48:40.778631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.778613    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:40.838852 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.838822    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"]
Oct 30 22:48:40.845625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.845597    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7e1fe5c2\""
Oct 30 22:48:40.864444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.864411    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqhx\" (UniqueName: \"kubernetes.io/projected/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-kube-api-access-vfqhx\") pod \"payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:40.864631 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.864518    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:40.886272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.886191    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7e1fe5c2-dockercfg-2jjcj\""
Oct 30 22:48:40.926251 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.926199    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"]
Oct 30 22:48:40.930200 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.930178    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:40.965469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.965397    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:40.965469 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.965439    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:40.965665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.965609    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqhx\" (UniqueName: \"kubernetes.io/projected/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-kube-api-access-vfqhx\") pod \"payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:40.965730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.965665    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbwn\" (UniqueName: \"kubernetes.io/projected/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-kube-api-access-cgbwn\") pod \"payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:40.966117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.966095    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:40.989815 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:40.989786    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"]
Oct 30 22:48:41.029868 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.029837    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqhx\" (UniqueName: \"kubernetes.io/projected/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-kube-api-access-vfqhx\") pod \"payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:41.066682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.066656    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbwn\" (UniqueName: \"kubernetes.io/projected/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-kube-api-access-cgbwn\") pod \"payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:41.066854 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.066710    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:41.067372 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.067348    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:41.092137 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.092113    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:41.130657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.130622    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbwn\" (UniqueName: \"kubernetes.io/projected/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-kube-api-access-cgbwn\") pod \"payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") " pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:41.243531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.243443    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:41.472298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.472270    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"]
Oct 30 22:48:41.473802 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:48:41.473777    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48346cd_1f6a_4841_9ff9_9aba7f5545d3.slice/crio-96fa45bfe4a9f320d48008d4d98f53e72d17f1d9fb54784bcc9b4117539d57a1 WatchSource:0}: Error finding container 96fa45bfe4a9f320d48008d4d98f53e72d17f1d9fb54784bcc9b4117539d57a1: Status 404 returned error can't find the container with id 96fa45bfe4a9f320d48008d4d98f53e72d17f1d9fb54784bcc9b4117539d57a1
Oct 30 22:48:41.624197 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.624163    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"]
Oct 30 22:48:41.626671 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:48:41.626632    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb90654d_6712_4fdd_b4dc_5169b7dbf6e5.slice/crio-2f4839a5d3959e66fb48c6d9aa6f99698d24c291420148cc74f6692930712f7b WatchSource:0}: Error finding container 2f4839a5d3959e66fb48c6d9aa6f99698d24c291420148cc74f6692930712f7b: Status 404 returned error can't find the container with id 2f4839a5d3959e66fb48c6d9aa6f99698d24c291420148cc74f6692930712f7b
Oct 30 22:48:41.661359 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.661328    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:41.699116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.699086    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229" event={"ID":"c48346cd-1f6a-4841-9ff9-9aba7f5545d3","Type":"ContainerStarted","Data":"48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93"}
Oct 30 22:48:41.699258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.699147    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229" event={"ID":"c48346cd-1f6a-4841-9ff9-9aba7f5545d3","Type":"ContainerStarted","Data":"96fa45bfe4a9f320d48008d4d98f53e72d17f1d9fb54784bcc9b4117539d57a1"}
Oct 30 22:48:41.699313 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.699304    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:41.700948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.700927    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr" event={"ID":"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5","Type":"ContainerStarted","Data":"2f4839a5d3959e66fb48c6d9aa6f99698d24c291420148cc74f6692930712f7b"}
Oct 30 22:48:41.819517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.819405    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229" podStartSLOduration=1.819392561 podStartE2EDuration="1.819392561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:41.819138252 +0000 UTC m=+3533.343281045" watchObservedRunningTime="2026-10-30 22:48:41.819392561 +0000 UTC m=+3533.343535322"
Oct 30 22:48:41.901866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:41.901832    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:42.709051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:42.709018    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr" event={"ID":"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5","Type":"ContainerStarted","Data":"67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f"}
Oct 30 22:48:42.709550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:42.709359    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:42.829848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:42.829797    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr" podStartSLOduration=2.829785435 podStartE2EDuration="2.829785435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:42.82878105 +0000 UTC m=+3534.352923836" watchObservedRunningTime="2026-10-30 22:48:42.829785435 +0000 UTC m=+3534.353928223"
Oct 30 22:48:45.467760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:45.467720    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:45.699351 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:45.699318    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:47.336743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:47.336686    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:50.183546 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:50.183502    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:48:50.578958 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:48:50.578865    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:48:51.062307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:51.062194    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"]
Oct 30 22:48:51.125237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:51.124243    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gskpz"]
Oct 30 22:48:52.429453 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:52.429389    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:52.648738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:52.648702    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2f5894-0f32-4c3b-8d69-cc6fda43f44b" path="/var/lib/kubelet/pods/7a2f5894-0f32-4c3b-8d69-cc6fda43f44b/volumes"
Oct 30 22:48:52.732242 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:52.732109    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:48:53.733424 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:53.733388    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:48:57.508932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:57.508835    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:59.550437 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:59.550354    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:59.816766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:59.816689    5031 generic.go:356] "Generic (PLEG): container finished" podID="7fee8d26-0788-4757-b949-acc1ae153f55" containerID="a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c" exitCode=0
Oct 30 22:48:59.816919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:59.816771    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" event={"ID":"7fee8d26-0788-4757-b949-acc1ae153f55","Type":"ContainerDied","Data":"a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c"}
Oct 30 22:48:59.817151 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:59.817136    5031 scope.go:122] "RemoveContainer" containerID="a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c"
Oct 30 22:48:59.818388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:59.818352    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:59.818472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:59.818416    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:59.818472 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:48:59.818448    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:00.835655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:00.835617    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" event={"ID":"7fee8d26-0788-4757-b949-acc1ae153f55","Type":"ContainerStarted","Data":"ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e"}
Oct 30 22:49:01.638283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:01.638254    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:49:01.639494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:01.639465    5031 patch_prober.go:29] interesting pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 22:49:01.639625 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:01.639524    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 22:49:02.639222 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:02.639168    5031 patch_prober.go:29] interesting pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 22:49:02.639628 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:02.639247    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 22:49:02.834056 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:02.834018    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:03.638554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:03.638513    5031 patch_prober.go:29] interesting pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 22:49:03.638742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:03.638571    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 22:49:04.638741 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:04.638705    5031 patch_prober.go:29] interesting pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 22:49:04.639170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:04.638761    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 22:49:05.638725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:05.638688    5031 patch_prober.go:29] interesting pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 30 22:49:05.638957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:05.638769    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 30 22:49:05.664345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:05.664306    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:49:06.646199 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:06.646171    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:49:06.647315 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:06.647297    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:49:08.372896 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:08.372862    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:12.979489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:12.979447    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:13.499894 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.499864    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:13.537271 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.537238    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"]
Oct 30 22:49:13.541145 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.541128    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.601229 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.601187    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"]
Oct 30 22:49:13.635132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.635097    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.635132 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.635130    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65p6q\" (UniqueName: \"kubernetes.io/projected/43b28165-8f4c-490d-93ff-2e4b719194aa-kube-api-access-65p6q\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.635432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.635307    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.648427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.648397    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:13.648427 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.648416    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:13.648588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.648430    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:13.648588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.648454    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfs8q\""
Oct 30 22:49:13.736431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.736391    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.736431 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.736427    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65p6q\" (UniqueName: \"kubernetes.io/projected/43b28165-8f4c-490d-93ff-2e4b719194aa-kube-api-access-65p6q\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.736713 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.736612    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.737083 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.737066    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.739042 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.739010    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.800878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.800805    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65p6q\" (UniqueName: \"kubernetes.io/projected/43b28165-8f4c-490d-93ff-2e4b719194aa-kube-api-access-65p6q\") pod \"maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:13.854554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:13.854509    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:14.238325 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:14.238299    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"]
Oct 30 22:49:14.239023 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:49:14.238990    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b28165_8f4c_490d_93ff_2e4b719194aa.slice/crio-2f1861ae0293942841a1da1d802e39170fcf2277838fd0c8956449d5de26482c WatchSource:0}: Error finding container 2f1861ae0293942841a1da1d802e39170fcf2277838fd0c8956449d5de26482c: Status 404 returned error can't find the container with id 2f1861ae0293942841a1da1d802e39170fcf2277838fd0c8956449d5de26482c
Oct 30 22:49:14.918517 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:14.918482    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd" event={"ID":"43b28165-8f4c-490d-93ff-2e4b719194aa","Type":"ContainerStarted","Data":"e88ce205aebf51eeb4d7ba3d34105b37097b2fea14c1575e23273743d15c1349"}
Oct 30 22:49:14.918676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:14.918541    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd" event={"ID":"43b28165-8f4c-490d-93ff-2e4b719194aa","Type":"ContainerStarted","Data":"2f1861ae0293942841a1da1d802e39170fcf2277838fd0c8956449d5de26482c"}
Oct 30 22:49:15.044553 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:15.044501    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd" podStartSLOduration=2.044484006 podStartE2EDuration="2.044484006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:15.042964938 +0000 UTC m=+3566.567107720" watchObservedRunningTime="2026-10-30 22:49:15.044484006 +0000 UTC m=+3566.568626769"
Oct 30 22:49:15.926641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:15.926604    5031 generic.go:356] "Generic (PLEG): container finished" podID="43b28165-8f4c-490d-93ff-2e4b719194aa" containerID="e88ce205aebf51eeb4d7ba3d34105b37097b2fea14c1575e23273743d15c1349" exitCode=0
Oct 30 22:49:15.927057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:15.926685    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd" event={"ID":"43b28165-8f4c-490d-93ff-2e4b719194aa","Type":"ContainerDied","Data":"e88ce205aebf51eeb4d7ba3d34105b37097b2fea14c1575e23273743d15c1349"}
Oct 30 22:49:17.066667 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.066642    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:17.170476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.170432    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-xks-ca\") pod \"43b28165-8f4c-490d-93ff-2e4b719194aa\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") "
Oct 30 22:49:17.170665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.170532    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-openshift-ca\") pod \"43b28165-8f4c-490d-93ff-2e4b719194aa\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") "
Oct 30 22:49:17.170665 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.170581    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b28165-8f4c-490d-93ff-2e4b719194aa-kube-api-access-65p6q\" (UniqueName: \"kubernetes.io/projected/43b28165-8f4c-490d-93ff-2e4b719194aa-kube-api-access-65p6q\") pod \"43b28165-8f4c-490d-93ff-2e4b719194aa\" (UID: \"43b28165-8f4c-490d-93ff-2e4b719194aa\") "
Oct 30 22:49:17.170935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.170912    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-openshift-ca" pod "43b28165-8f4c-490d-93ff-2e4b719194aa" (UID: "43b28165-8f4c-490d-93ff-2e4b719194aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:17.172842 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.172819    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b28165-8f4c-490d-93ff-2e4b719194aa-kube-api-access-65p6q" pod "43b28165-8f4c-490d-93ff-2e4b719194aa" (UID: "43b28165-8f4c-490d-93ff-2e4b719194aa"). InnerVolumeSpecName "kube-api-access-65p6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:17.172842 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.172830    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-xks-ca" pod "43b28165-8f4c-490d-93ff-2e4b719194aa" (UID: "43b28165-8f4c-490d-93ff-2e4b719194aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:17.271962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.271878    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.271962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.271913    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65p6q\" (UniqueName: \"kubernetes.io/projected/43b28165-8f4c-490d-93ff-2e4b719194aa-kube-api-access-65p6q\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.271962 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.271922    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b28165-8f4c-490d-93ff-2e4b719194aa-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.940459 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.940429    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"
Oct 30 22:49:17.940654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.940464    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd" event={"ID":"43b28165-8f4c-490d-93ff-2e4b719194aa","Type":"ContainerDied","Data":"2f1861ae0293942841a1da1d802e39170fcf2277838fd0c8956449d5de26482c"}
Oct 30 22:49:17.940654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:17.940516    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1861ae0293942841a1da1d802e39170fcf2277838fd0c8956449d5de26482c"
Oct 30 22:49:18.695389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:18.695357    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:20.540812 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:49:20.540772    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:49:23.852309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:23.852275    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:23.931513 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:23.931484    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"]
Oct 30 22:49:24.042822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:24.042791    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"]
Oct 30 22:49:24.373626 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:24.373595    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:24.603476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:24.603443    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:28.911769 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:28.911683    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:28.932564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:28.932519    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr" podUID="eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" containerName="payload-processing" containerID="cri-o://67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f" gracePeriod=25
Oct 30 22:49:29.043498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.043466    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229" podUID="c48346cd-1f6a-4841-9ff9-9aba7f5545d3" containerName="payload-pre-processing" containerID="cri-o://48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93" gracePeriod=25
Oct 30 22:49:29.192324 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.192289    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:49:29.288650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.288625    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-plugins-config-volume\") pod \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") "
Oct 30 22:49:29.288859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.288699    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-kube-api-access-cgbwn\" (UniqueName: \"kubernetes.io/projected/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-kube-api-access-cgbwn\") pod \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\" (UID: \"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5\") "
Oct 30 22:49:29.289055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.289031    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-plugins-config-volume" pod "eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" (UID: "eb90654d-6712-4fdd-b4dc-5169b7dbf6e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:29.289183 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.289166    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:29.291175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.291151    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-kube-api-access-cgbwn" pod "eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" (UID: "eb90654d-6712-4fdd-b4dc-5169b7dbf6e5"). InnerVolumeSpecName "kube-api-access-cgbwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:29.318175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.318149    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:49:29.389997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.389970    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-plugins-config-volume\") pod \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") "
Oct 30 22:49:29.390120 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.390041    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-kube-api-access-vfqhx\" (UniqueName: \"kubernetes.io/projected/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-kube-api-access-vfqhx\") pod \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\" (UID: \"c48346cd-1f6a-4841-9ff9-9aba7f5545d3\") "
Oct 30 22:49:29.390253 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.390238    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgbwn\" (UniqueName: \"kubernetes.io/projected/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5-kube-api-access-cgbwn\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:29.390434 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.390409    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-plugins-config-volume" pod "c48346cd-1f6a-4841-9ff9-9aba7f5545d3" (UID: "c48346cd-1f6a-4841-9ff9-9aba7f5545d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:29.392404 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.392382    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-kube-api-access-vfqhx" pod "c48346cd-1f6a-4841-9ff9-9aba7f5545d3" (UID: "c48346cd-1f6a-4841-9ff9-9aba7f5545d3"). InnerVolumeSpecName "kube-api-access-vfqhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:29.491770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.491694    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfqhx\" (UniqueName: \"kubernetes.io/projected/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-kube-api-access-vfqhx\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:29.491770 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:29.491720    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c48346cd-1f6a-4841-9ff9-9aba7f5545d3-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:30.014274 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.014244    5031 generic.go:356] "Generic (PLEG): container finished" podID="c48346cd-1f6a-4841-9ff9-9aba7f5545d3" containerID="48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93" exitCode=0
Oct 30 22:49:30.014712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.014291    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"
Oct 30 22:49:30.014712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.014326    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229" event={"ID":"c48346cd-1f6a-4841-9ff9-9aba7f5545d3","Type":"ContainerDied","Data":"48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93"}
Oct 30 22:49:30.014712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.014375    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229" event={"ID":"c48346cd-1f6a-4841-9ff9-9aba7f5545d3","Type":"ContainerDied","Data":"96fa45bfe4a9f320d48008d4d98f53e72d17f1d9fb54784bcc9b4117539d57a1"}
Oct 30 22:49:30.014712 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.014405    5031 scope.go:122] "RemoveContainer" containerID="48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93"
Oct 30 22:49:30.015687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.015668    5031 generic.go:356] "Generic (PLEG): container finished" podID="eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" containerID="67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f" exitCode=0
Oct 30 22:49:30.015763 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.015719    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"
Oct 30 22:49:30.015836 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.015761    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr" event={"ID":"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5","Type":"ContainerDied","Data":"67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f"}
Oct 30 22:49:30.015889 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.015839    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr" event={"ID":"eb90654d-6712-4fdd-b4dc-5169b7dbf6e5","Type":"ContainerDied","Data":"2f4839a5d3959e66fb48c6d9aa6f99698d24c291420148cc74f6692930712f7b"}
Oct 30 22:49:30.026686 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.026136    5031 scope.go:122] "RemoveContainer" containerID="48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93"
Oct 30 22:49:30.027380 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:49:30.027349    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93\": container with ID starting with 48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93 not found: ID does not exist" containerID="48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93"
Oct 30 22:49:30.027492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.027390    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93"} err="failed to get container status \"48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93\": rpc error: code = NotFound desc = could not find container \"48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93\": container with ID starting with 48b7d6364565857038925de0dae2878ff804a0de849f832d4192e1face845c93 not found: ID does not exist"
Oct 30 22:49:30.027492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.027415    5031 scope.go:122] "RemoveContainer" containerID="67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f"
Oct 30 22:49:30.036127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.036109    5031 scope.go:122] "RemoveContainer" containerID="67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f"
Oct 30 22:49:30.036390 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:49:30.036371    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f\": container with ID starting with 67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f not found: ID does not exist" containerID="67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f"
Oct 30 22:49:30.036443 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.036398    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f"} err="failed to get container status \"67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f\": rpc error: code = NotFound desc = could not find container \"67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f\": container with ID starting with 67a7b2b9e3f57b3ea71f5b1744fb73a37114d5e8610adc4ac51b6c882cc8d76f not found: ID does not exist"
Oct 30 22:49:30.184520 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.184490    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"]
Oct 30 22:49:30.190464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.190440    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr"]
Oct 30 22:49:30.304115 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.304033    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"]
Oct 30 22:49:30.306616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.306595    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229"]
Oct 30 22:49:30.651564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.651530    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48346cd-1f6a-4841-9ff9-9aba7f5545d3" path="/var/lib/kubelet/pods/c48346cd-1f6a-4841-9ff9-9aba7f5545d3/volumes"
Oct 30 22:49:30.651932 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:30.651918    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" path="/var/lib/kubelet/pods/eb90654d-6712-4fdd-b4dc-5169b7dbf6e5/volumes"
Oct 30 22:49:33.341142 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:33.341106    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:34.917470 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:34.917442    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:35.229399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:35.229320    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:35.397114 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:35.397082    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"]
Oct 30 22:49:35.448296 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:35.448270    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e1fe5c2-vwxmd"]
Oct 30 22:49:36.644322 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:36.644287    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b28165-8f4c-490d-93ff-2e4b719194aa" path="/var/lib/kubelet/pods/43b28165-8f4c-490d-93ff-2e4b719194aa/volumes"
Oct 30 22:49:37.889582 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:37.889542    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:49:39.822832 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:39.822797    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"]
Oct 30 22:49:39.823293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:39.823009    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy" containerID="cri-o://ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e" gracePeriod=30
Oct 30 22:49:39.901529 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:39.901495    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:41.170429 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.170396    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:41.642952 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.642921    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"]
Oct 30 22:49:41.643399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643379    5031 cpu_manager.go:414] "containerMap: removing container" podUID="43b28165-8f4c-490d-93ff-2e4b719194aa" containerName="revoke-keys"
Oct 30 22:49:41.643399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643395    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b28165-8f4c-490d-93ff-2e4b719194aa" containerName="revoke-keys"
Oct 30 22:49:41.643610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643421    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c48346cd-1f6a-4841-9ff9-9aba7f5545d3" containerName="payload-pre-processing"
Oct 30 22:49:41.643610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643426    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48346cd-1f6a-4841-9ff9-9aba7f5545d3" containerName="payload-pre-processing"
Oct 30 22:49:41.643610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643433    5031 cpu_manager.go:414] "containerMap: removing container" podUID="eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" containerName="payload-processing"
Oct 30 22:49:41.643610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643438    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" containerName="payload-processing"
Oct 30 22:49:41.643610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643526    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b28165-8f4c-490d-93ff-2e4b719194aa" containerName="revoke-keys"
Oct 30 22:49:41.643610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643535    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48346cd-1f6a-4841-9ff9-9aba7f5545d3" containerName="payload-pre-processing"
Oct 30 22:49:41.643610 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.643541    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb90654d-6712-4fdd-b4dc-5169b7dbf6e5" containerName="payload-processing"
Oct 30 22:49:41.648291 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.648272    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.708768 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.708742    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-40b9a8-openshift-default-dockercfg-96dsc\""
Oct 30 22:49:41.756375 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.756340    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"]
Oct 30 22:49:41.806478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806445    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc6ef3-4865-462d-ac04-406d4a717302-istiod-ca-cert\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.806478 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806483    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-envoy\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.806725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806505    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-podinfo\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.806725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806568    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-socket\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.806725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806662    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-certs\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.806725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806710    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-data\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.806918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806732    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-token\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.806918 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806794    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558v9\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-kube-api-access-558v9\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.807034 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.806980    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-credential-socket\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.871725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.871694    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:41.908458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908364    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc6ef3-4865-462d-ac04-406d4a717302-istiod-ca-cert\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.908458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908434    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-envoy\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.908710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908472    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-podinfo\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.908710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908503    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-socket\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.908710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908536    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-certs\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.908710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908609    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-data\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.908710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908649    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-token\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.908710 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908676    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558v9\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-kube-api-access-558v9\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.909077 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.908775    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-credential-socket\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.909267 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.909239    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-credential-socket\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.910303 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.909710    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-socket\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.910519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.910332    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc6ef3-4865-462d-ac04-406d4a717302-istiod-ca-cert\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.910821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.910799    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-certs\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.911175 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.911148    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-data\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.915307 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.915282    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-podinfo\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.915655 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.915636    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-envoy\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.976353 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.976322    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-558v9\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-kube-api-access-558v9\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.976492 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.976434    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-token\") pod \"e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") " pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:41.980340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:41.980326    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:42.111116 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:42.111087    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:42.412580 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:42.412554    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"]
Oct 30 22:49:42.413445 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:49:42.413418    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fc6ef3_4865_462d_ac04_406d4a717302.slice/crio-c4c8db9519403d8e35353b9639536a0352fa6d36d0500a5ca66fb09af36bc2e1 WatchSource:0}: Error finding container c4c8db9519403d8e35353b9639536a0352fa6d36d0500a5ca66fb09af36bc2e1: Status 404 returned error can't find the container with id c4c8db9519403d8e35353b9639536a0352fa6d36d0500a5ca66fb09af36bc2e1
Oct 30 22:49:42.415706 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:42.415673    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:42.415783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:42.415738    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:42.415783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:42.415777    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:43.105835 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:43.105798    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" event={"ID":"b1fc6ef3-4865-462d-ac04-406d4a717302","Type":"ContainerStarted","Data":"f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7"}
Oct 30 22:49:43.106048 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:43.105856    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" event={"ID":"b1fc6ef3-4865-462d-ac04-406d4a717302","Type":"ContainerStarted","Data":"c4c8db9519403d8e35353b9639536a0352fa6d36d0500a5ca66fb09af36bc2e1"}
Oct 30 22:49:43.231036 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:43.230983    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podStartSLOduration=2.230971035 podStartE2EDuration="2.230971035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:43.230361411 +0000 UTC m=+3594.754504205" watchObservedRunningTime="2026-10-30 22:49:43.230971035 +0000 UTC m=+3594.755113798"
Oct 30 22:49:43.981279 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:43.981249    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:43.986340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:43.986315    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:44.114936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:44.114903    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:44.115925 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:44.115905    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:49:45.092500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.092478    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:49:45.123500 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.123466    5031 generic.go:356] "Generic (PLEG): container finished" podID="7fee8d26-0788-4757-b949-acc1ae153f55" containerID="ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e" exitCode=0
Oct 30 22:49:45.123641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.123575    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"
Oct 30 22:49:45.123641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.123605    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" event={"ID":"7fee8d26-0788-4757-b949-acc1ae153f55","Type":"ContainerDied","Data":"ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e"}
Oct 30 22:49:45.123725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.123660    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz" event={"ID":"7fee8d26-0788-4757-b949-acc1ae153f55","Type":"ContainerDied","Data":"641a6fc2f9fcc2891ce77545a891d5cfe33e54e24bebae56b5f968aa4ded2a5a"}
Oct 30 22:49:45.123725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.123691    5031 scope.go:122] "RemoveContainer" containerID="ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e"
Oct 30 22:49:45.136247 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.136200    5031 scope.go:122] "RemoveContainer" containerID="a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c"
Oct 30 22:49:45.145579 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.145557    5031 scope.go:122] "RemoveContainer" containerID="ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e"
Oct 30 22:49:45.145910 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:49:45.145884    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e\": container with ID starting with ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e not found: ID does not exist" containerID="ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e"
Oct 30 22:49:45.145970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.145924    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e"} err="failed to get container status \"ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e\": rpc error: code = NotFound desc = could not find container \"ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e\": container with ID starting with ae10405ff3426dd8dcd7562ec44e9da4ea118ba9e315e8e2bdd401e9334f211e not found: ID does not exist"
Oct 30 22:49:45.145970 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.145948    5031 scope.go:122] "RemoveContainer" containerID="a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c"
Oct 30 22:49:45.146268 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:49:45.146238    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c\": container with ID starting with a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c not found: ID does not exist" containerID="a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c"
Oct 30 22:49:45.146368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.146276    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c"} err="failed to get container status \"a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c\": rpc error: code = NotFound desc = could not find container \"a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c\": container with ID starting with a375da5cf477c4b4609b0d81d4699d3d3550f7dd25c1c1566a7d06fba3df407c not found: ID does not exist"
Oct 30 22:49:45.242129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242095    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-envoy\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242165    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fee8d26-0788-4757-b949-acc1ae153f55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fee8d26-0788-4757-b949-acc1ae153f55-istio-podinfo\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242197    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-credential-socket\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242282 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242243    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-kube-api-access-bdfdr\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-kube-api-access-bdfdr\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242288    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-istio-token\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-istio-token\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242365    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-certs\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242397    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-data\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242436    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-socket\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242509    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fee8d26-0788-4757-b949-acc1ae153f55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fee8d26-0788-4757-b949-acc1ae153f55-istiod-ca-cert\") pod \"7fee8d26-0788-4757-b949-acc1ae153f55\" (UID: \"7fee8d26-0788-4757-b949-acc1ae153f55\") "
Oct 30 22:49:45.242682 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242539    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-credential-socket" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:45.242792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242742    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-certs" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:45.242792 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.242755    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-data" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:45.243074 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.243051    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fee8d26-0788-4757-b949-acc1ae153f55-istiod-ca-cert" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:45.243249 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.243089    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-socket" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:45.243374 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.243201    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.243511 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.243488    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.243607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.243515    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.245121 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.245099    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fee8d26-0788-4757-b949-acc1ae153f55-istio-podinfo" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:45.245370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.245351    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-kube-api-access-bdfdr" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "kube-api-access-bdfdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:45.245575 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.245554    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-envoy" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:45.245926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.245901    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-istio-token" pod "7fee8d26-0788-4757-b949-acc1ae153f55" (UID: "7fee8d26-0788-4757-b949-acc1ae153f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:45.345049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.344967    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fee8d26-0788-4757-b949-acc1ae153f55-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.345049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.344992    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdfdr\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-kube-api-access-bdfdr\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.345049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.345002    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fee8d26-0788-4757-b949-acc1ae153f55-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.345049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.345012    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.345049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.345020    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fee8d26-0788-4757-b949-acc1ae153f55-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.345049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.345030    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fee8d26-0788-4757-b949-acc1ae153f55-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.552320 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.552280    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"]
Oct 30 22:49:45.559730 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:45.559703    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz"]
Oct 30 22:49:46.148444 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:46.148395    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.448960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:46.448875    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:46.650913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:46.650856    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" path="/var/lib/kubelet/pods/7fee8d26-0788-4757-b949-acc1ae153f55/volumes"
Oct 30 22:49:49.442376 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.442338    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:49.663555 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.663512    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:49.807029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.806950    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"]
Oct 30 22:49:49.807641 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.807621    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy"
Oct 30 22:49:49.807702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.807642    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy"
Oct 30 22:49:49.807702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.807667    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy"
Oct 30 22:49:49.807702 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.807673    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy"
Oct 30 22:49:49.807838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.807814    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy"
Oct 30 22:49:49.807838 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.807824    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fee8d26-0788-4757-b949-acc1ae153f55" containerName="istio-proxy"
Oct 30 22:49:49.812611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.812590    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:49.869620 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.869590    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"]
Oct 30 22:49:49.917334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.917302    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-40b9a8\""
Oct 30 22:49:49.917334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.917302    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-40b9a8-dockercfg-fhxdd\""
Oct 30 22:49:49.936611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.936581    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"]
Oct 30 22:49:49.940370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.940351    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:49.997743 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:49.997710    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"]
Oct 30 22:49:50.003257 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.003176    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61f736f-bc03-457f-b47b-d65dfc7fcadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:50.003403 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.003327    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbx8\" (UniqueName: \"kubernetes.io/projected/c61f736f-bc03-457f-b47b-d65dfc7fcadf-kube-api-access-5cbx8\") pod \"payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:50.104611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.104582    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61f736f-bc03-457f-b47b-d65dfc7fcadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:50.104750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.104678    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbx8\" (UniqueName: \"kubernetes.io/projected/c61f736f-bc03-457f-b47b-d65dfc7fcadf-kube-api-access-5cbx8\") pod \"payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:50.104810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.104758    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5f7s\" (UniqueName: \"kubernetes.io/projected/05df6d89-3723-4449-b11d-8a2081237bcc-kube-api-access-t5f7s\") pod \"payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:50.104867 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.104840    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05df6d89-3723-4449-b11d-8a2081237bcc-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:50.105432 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.105415    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61f736f-bc03-457f-b47b-d65dfc7fcadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:50.143611 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.143590    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:49:50.166118 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.166098    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:49:50.168701 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.168679    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbx8\" (UniqueName: \"kubernetes.io/projected/c61f736f-bc03-457f-b47b-d65dfc7fcadf-kube-api-access-5cbx8\") pod \"payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:50.206436 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.206408    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5f7s\" (UniqueName: \"kubernetes.io/projected/05df6d89-3723-4449-b11d-8a2081237bcc-kube-api-access-t5f7s\") pod \"payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:50.206572 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.206460    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05df6d89-3723-4449-b11d-8a2081237bcc-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:50.207046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.207028    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05df6d89-3723-4449-b11d-8a2081237bcc-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:50.269996 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.269959    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5f7s\" (UniqueName: \"kubernetes.io/projected/05df6d89-3723-4449-b11d-8a2081237bcc-kube-api-access-t5f7s\") pod \"payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:50.425848 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.425760    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:50.550888 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:49:50.550852    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:49:50.553439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.553417    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:50.814897 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:49:50.814870    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61f736f_bc03_457f_b47b_d65dfc7fcadf.slice/crio-a0184d85fe9a7daa7eab5cccedfdb4e8e8708a9d0227d979db1945c5ea46341d WatchSource:0}: Error finding container a0184d85fe9a7daa7eab5cccedfdb4e8e8708a9d0227d979db1945c5ea46341d: Status 404 returned error can't find the container with id a0184d85fe9a7daa7eab5cccedfdb4e8e8708a9d0227d979db1945c5ea46341d
Oct 30 22:49:50.815055 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.815004    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"]
Oct 30 22:49:50.933849 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:49:50.933817    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05df6d89_3723_4449_b11d_8a2081237bcc.slice/crio-68cc4b74cb17c9254b533b9149d05a1e0bda0077c19dba433f70e3435d61a4e2 WatchSource:0}: Error finding container 68cc4b74cb17c9254b533b9149d05a1e0bda0077c19dba433f70e3435d61a4e2: Status 404 returned error can't find the container with id 68cc4b74cb17c9254b533b9149d05a1e0bda0077c19dba433f70e3435d61a4e2
Oct 30 22:49:50.934126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:50.934104    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"]
Oct 30 22:49:51.179679 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.179632    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw" event={"ID":"05df6d89-3723-4449-b11d-8a2081237bcc","Type":"ContainerStarted","Data":"bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae"}
Oct 30 22:49:51.179873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.179708    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw" event={"ID":"05df6d89-3723-4449-b11d-8a2081237bcc","Type":"ContainerStarted","Data":"68cc4b74cb17c9254b533b9149d05a1e0bda0077c19dba433f70e3435d61a4e2"}
Oct 30 22:49:51.179873 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.179781    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:49:51.181532 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.181506    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt" event={"ID":"c61f736f-bc03-457f-b47b-d65dfc7fcadf","Type":"ContainerStarted","Data":"d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4"}
Oct 30 22:49:51.181661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.181544    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt" event={"ID":"c61f736f-bc03-457f-b47b-d65dfc7fcadf","Type":"ContainerStarted","Data":"a0184d85fe9a7daa7eab5cccedfdb4e8e8708a9d0227d979db1945c5ea46341d"}
Oct 30 22:49:51.181661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.181647    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:49:51.300507 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.300460    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw" podStartSLOduration=2.300447844 podStartE2EDuration="2.300447844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:51.298954279 +0000 UTC m=+3602.823097054" watchObservedRunningTime="2026-10-30 22:49:51.300447844 +0000 UTC m=+3602.824590607"
Oct 30 22:49:51.424462 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:51.423003    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt" podStartSLOduration=2.422982033 podStartE2EDuration="2.422982033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:51.418705696 +0000 UTC m=+3602.942848512" watchObservedRunningTime="2026-10-30 22:49:51.422982033 +0000 UTC m=+3602.947124794"
Oct 30 22:49:52.746148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:52.746090    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.966471 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:52.966441    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.998796 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:52.998720    5031 scope.go:122] "RemoveContainer" containerID="49efb01a2b4371951da36e4693d8fa37283950bf1eff72fbfce28c9db3a21890"
Oct 30 22:49:53.008430 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:53.008407    5031 scope.go:122] "RemoveContainer" containerID="8674c5ed0b1887e95b7cc7fe0d7fdbfe05503a2c29abfd091bd59b395363f847"
Oct 30 22:49:53.192024 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:53.191974    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:57.910936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:57.910904    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:59.910599 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:49:59.910569    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:01.042195 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:01.042155    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:50:02.190108 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:02.190080    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:50:02.190590 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:02.190145    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:50:03.290766 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:03.290738    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:08.102337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:08.102304    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.332982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:10.332896    5031 generic.go:356] "Generic (PLEG): container finished" podID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerID="f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7" exitCode=0
Oct 30 22:50:10.332982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:10.332969    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" event={"ID":"b1fc6ef3-4865-462d-ac04-406d4a717302","Type":"ContainerDied","Data":"f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7"}
Oct 30 22:50:10.333466 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:10.333353    5031 scope.go:122] "RemoveContainer" containerID="f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7"
Oct 30 22:50:10.334547 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:10.334515    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:50:10.334669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:10.334600    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:50:10.334669 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:10.334651    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:50:10.932716 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:10.932681    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:11.341419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:11.341329    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" event={"ID":"b1fc6ef3-4865-462d-ac04-406d4a717302","Type":"ContainerStarted","Data":"68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd"}
Oct 30 22:50:11.980855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:11.980819    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:50:11.981869 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:11.981847    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 22:50:11.981927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:11.981896    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 22:50:12.980821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:12.980783    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 22:50:12.981284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:12.980843    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 22:50:13.485600 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:13.485562    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:13.981694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:13.981657    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 22:50:13.982144 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:13.981716    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 22:50:14.118338 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:14.118301    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:50:14.981377 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:14.981341    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 22:50:14.981556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:14.981405    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 22:50:15.981753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:15.981712    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 22:50:15.982250 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:15.981779    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 22:50:16.984995 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:16.984963    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:50:16.986019 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:16.986002    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:50:18.509570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:18.509540    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:20.539933 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:50:20.539890    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:50:23.059177 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:23.059143    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"]
Oct 30 22:50:23.109738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:23.109713    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d21d9-rwz5j"]
Oct 30 22:50:23.614181 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:23.614148    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:24.644841 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:24.644806    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cc876a-0246-4b93-8c3d-e385896958b7" path="/var/lib/kubelet/pods/18cc876a-0246-4b93-8c3d-e385896958b7/volumes"
Oct 30 22:50:28.611531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:28.611447    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:33.701489 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:33.701451    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:38.802687 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:38.802607    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:41.151543 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:41.151505    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:41.373685 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:41.373652    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:43.901033 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:43.901000    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:48.899790 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:48.899757    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:50.540239 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:50:50.540193    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:50:54.024333 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:54.024301    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:59.060728 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:50:59.060696    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:00.262519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:00.262482    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:00.558824 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:00.558734    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:04.197518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:04.197480    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:09.118373 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:09.118343    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:14.200697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:14.200669    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:19.196421 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:19.196386    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:20.540631 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:51:20.540589    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:51:24.354110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:24.354076    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:29.328039 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:29.327958    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:31.540265 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.540228    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"]
Oct 30 22:51:31.544030 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.544013    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.598753 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.598721    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:31.629148 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.629113    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.629335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.629255    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.629387 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.629349    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jll58\" (UniqueName: \"kubernetes.io/projected/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-kube-api-access-jll58\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.648697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.648672    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:31.648697 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.648678    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:31.649021 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.648707    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfs8q\""
Oct 30 22:51:31.652011 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.651993    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"]
Oct 30 22:51:31.729928 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.729892    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jll58\" (UniqueName: \"kubernetes.io/projected/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-kube-api-access-jll58\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.730131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.729979    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.730131 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.730060    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.730742 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.730719    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.732588 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.732566    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.793661 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.793593    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jll58\" (UniqueName: \"kubernetes.io/projected/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-kube-api-access-jll58\") pod \"maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:31.857288 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:31.857238    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:32.235152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:32.235122    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"]
Oct 30 22:51:32.236836 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:51:32.236808    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3b94c0_ecbe_46cf_b1fd_2b37931edc16.slice/crio-9ad6e40c8e8eeda5c9cc4f5d0e6c5dc01a5e566d8ec12c3c1c7a4e437bafdfee WatchSource:0}: Error finding container 9ad6e40c8e8eeda5c9cc4f5d0e6c5dc01a5e566d8ec12c3c1c7a4e437bafdfee: Status 404 returned error can't find the container with id 9ad6e40c8e8eeda5c9cc4f5d0e6c5dc01a5e566d8ec12c3c1c7a4e437bafdfee
Oct 30 22:51:32.842732 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:32.842706    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62" event={"ID":"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16","Type":"ContainerStarted","Data":"a8b58d39c14467f9bbb6da1b4d8f168becc084d0f52498f274a6f7ce62ecd18f"}
Oct 30 22:51:32.843127 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:32.842755    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62" event={"ID":"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16","Type":"ContainerStarted","Data":"9ad6e40c8e8eeda5c9cc4f5d0e6c5dc01a5e566d8ec12c3c1c7a4e437bafdfee"}
Oct 30 22:51:32.964337 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:32.964273    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62" podStartSLOduration=1.96425435 podStartE2EDuration="1.96425435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:32.962501035 +0000 UTC m=+3704.486643818" watchObservedRunningTime="2026-10-30 22:51:32.96425435 +0000 UTC m=+3704.488397116"
Oct 30 22:51:33.850540 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:33.850503    5031 generic.go:356] "Generic (PLEG): container finished" podID="ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" containerID="a8b58d39c14467f9bbb6da1b4d8f168becc084d0f52498f274a6f7ce62ecd18f" exitCode=0
Oct 30 22:51:33.850975 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:33.850594    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62" event={"ID":"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16","Type":"ContainerDied","Data":"a8b58d39c14467f9bbb6da1b4d8f168becc084d0f52498f274a6f7ce62ecd18f"}
Oct 30 22:51:34.695290 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:34.695256    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:34.994319 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:34.994289    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:35.061309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.061269    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-xks-ca\") pod \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") "
Oct 30 22:51:35.061509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.061355    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-openshift-ca\") pod \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") "
Oct 30 22:51:35.061509 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.061400    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-kube-api-access-jll58\" (UniqueName: \"kubernetes.io/projected/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-kube-api-access-jll58\") pod \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\" (UID: \"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16\") "
Oct 30 22:51:35.061757 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.061727    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-openshift-ca" pod "ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" (UID: "ce3b94c0-ecbe-46cf-b1fd-2b37931edc16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:35.063666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.063639    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-xks-ca" pod "ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" (UID: "ce3b94c0-ecbe-46cf-b1fd-2b37931edc16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:35.063666 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.063648    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-kube-api-access-jll58" pod "ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" (UID: "ce3b94c0-ecbe-46cf-b1fd-2b37931edc16"). InnerVolumeSpecName "kube-api-access-jll58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:35.162783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.162735    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:35.162783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.162777    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jll58\" (UniqueName: \"kubernetes.io/projected/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-kube-api-access-jll58\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:35.162783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.162791    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:35.863672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.863640    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"
Oct 30 22:51:35.863672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.863662    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62" event={"ID":"ce3b94c0-ecbe-46cf-b1fd-2b37931edc16","Type":"ContainerDied","Data":"9ad6e40c8e8eeda5c9cc4f5d0e6c5dc01a5e566d8ec12c3c1c7a4e437bafdfee"}
Oct 30 22:51:35.863913 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:35.863708    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad6e40c8e8eeda5c9cc4f5d0e6c5dc01a5e566d8ec12c3c1c7a4e437bafdfee"
Oct 30 22:51:39.721345 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:39.721310    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:41.921664 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:41.921576    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"]
Oct 30 22:51:42.032725 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:42.032692    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"]
Oct 30 22:51:42.363336 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:42.363184    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:42.602101 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:42.602069    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:46.922298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:46.922251    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw" podUID="05df6d89-3723-4449-b11d-8a2081237bcc" containerName="payload-processing" containerID="cri-o://bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae" gracePeriod=25
Oct 30 22:51:47.033005 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.032956    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt" podUID="c61f736f-bc03-457f-b47b-d65dfc7fcadf" containerName="payload-pre-processing" containerID="cri-o://d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4" gracePeriod=25
Oct 30 22:51:47.236280 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.236238    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:51:47.341053 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.341023    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:51:47.392597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.392558    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05df6d89-3723-4449-b11d-8a2081237bcc-kube-api-access-t5f7s\" (UniqueName: \"kubernetes.io/projected/05df6d89-3723-4449-b11d-8a2081237bcc-kube-api-access-t5f7s\") pod \"05df6d89-3723-4449-b11d-8a2081237bcc\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") "
Oct 30 22:51:47.392818 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.392613    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05df6d89-3723-4449-b11d-8a2081237bcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05df6d89-3723-4449-b11d-8a2081237bcc-plugins-config-volume\") pod \"05df6d89-3723-4449-b11d-8a2081237bcc\" (UID: \"05df6d89-3723-4449-b11d-8a2081237bcc\") "
Oct 30 22:51:47.393408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.393380    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05df6d89-3723-4449-b11d-8a2081237bcc-plugins-config-volume" pod "05df6d89-3723-4449-b11d-8a2081237bcc" (UID: "05df6d89-3723-4449-b11d-8a2081237bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:47.395527 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.395497    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05df6d89-3723-4449-b11d-8a2081237bcc-kube-api-access-t5f7s" pod "05df6d89-3723-4449-b11d-8a2081237bcc" (UID: "05df6d89-3723-4449-b11d-8a2081237bcc"). InnerVolumeSpecName "kube-api-access-t5f7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:47.493755 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.493647    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61f736f-bc03-457f-b47b-d65dfc7fcadf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61f736f-bc03-457f-b47b-d65dfc7fcadf-plugins-config-volume\") pod \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") "
Oct 30 22:51:47.493926 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.493840    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61f736f-bc03-457f-b47b-d65dfc7fcadf-kube-api-access-5cbx8\" (UniqueName: \"kubernetes.io/projected/c61f736f-bc03-457f-b47b-d65dfc7fcadf-kube-api-access-5cbx8\") pod \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\" (UID: \"c61f736f-bc03-457f-b47b-d65dfc7fcadf\") "
Oct 30 22:51:47.494126 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.494098    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61f736f-bc03-457f-b47b-d65dfc7fcadf-plugins-config-volume" pod "c61f736f-bc03-457f-b47b-d65dfc7fcadf" (UID: "c61f736f-bc03-457f-b47b-d65dfc7fcadf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:47.494196 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.494177    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5f7s\" (UniqueName: \"kubernetes.io/projected/05df6d89-3723-4449-b11d-8a2081237bcc-kube-api-access-t5f7s\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:47.494272 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.494223    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05df6d89-3723-4449-b11d-8a2081237bcc-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:47.496310 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.496283    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61f736f-bc03-457f-b47b-d65dfc7fcadf-kube-api-access-5cbx8" pod "c61f736f-bc03-457f-b47b-d65dfc7fcadf" (UID: "c61f736f-bc03-457f-b47b-d65dfc7fcadf"). InnerVolumeSpecName "kube-api-access-5cbx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:47.595158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.595115    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61f736f-bc03-457f-b47b-d65dfc7fcadf-plugins-config-volume\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:47.595158 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.595152    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbx8\" (UniqueName: \"kubernetes.io/projected/c61f736f-bc03-457f-b47b-d65dfc7fcadf-kube-api-access-5cbx8\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:47.939051 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.939020    5031 generic.go:356] "Generic (PLEG): container finished" podID="05df6d89-3723-4449-b11d-8a2081237bcc" containerID="bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae" exitCode=0
Oct 30 22:51:47.939514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.939073    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"
Oct 30 22:51:47.939514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.939103    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw" event={"ID":"05df6d89-3723-4449-b11d-8a2081237bcc","Type":"ContainerDied","Data":"bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae"}
Oct 30 22:51:47.939514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.939154    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw" event={"ID":"05df6d89-3723-4449-b11d-8a2081237bcc","Type":"ContainerDied","Data":"68cc4b74cb17c9254b533b9149d05a1e0bda0077c19dba433f70e3435d61a4e2"}
Oct 30 22:51:47.939514 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.939175    5031 scope.go:122] "RemoveContainer" containerID="bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae"
Oct 30 22:51:47.940866 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.940846    5031 generic.go:356] "Generic (PLEG): container finished" podID="c61f736f-bc03-457f-b47b-d65dfc7fcadf" containerID="d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4" exitCode=0
Oct 30 22:51:47.940990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.940945    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt" event={"ID":"c61f736f-bc03-457f-b47b-d65dfc7fcadf","Type":"ContainerDied","Data":"d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4"}
Oct 30 22:51:47.940990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.940956    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"
Oct 30 22:51:47.940990 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.940979    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt" event={"ID":"c61f736f-bc03-457f-b47b-d65dfc7fcadf","Type":"ContainerDied","Data":"a0184d85fe9a7daa7eab5cccedfdb4e8e8708a9d0227d979db1945c5ea46341d"}
Oct 30 22:51:47.950389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.950369    5031 scope.go:122] "RemoveContainer" containerID="bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae"
Oct 30 22:51:47.950686 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:51:47.950658    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae\": container with ID starting with bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae not found: ID does not exist" containerID="bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae"
Oct 30 22:51:47.950808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.950691    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae"} err="failed to get container status \"bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae\": rpc error: code = NotFound desc = could not find container \"bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae\": container with ID starting with bdb20f16739d812543b96aeb8cc95d79f6945e40dd28c0dc589cf41becc742ae not found: ID does not exist"
Oct 30 22:51:47.950808 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.950715    5031 scope.go:122] "RemoveContainer" containerID="d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4"
Oct 30 22:51:47.960692 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.960673    5031 scope.go:122] "RemoveContainer" containerID="d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4"
Oct 30 22:51:47.960971 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:51:47.960951    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4\": container with ID starting with d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4 not found: ID does not exist" containerID="d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4"
Oct 30 22:51:47.961057 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:47.960979    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4"} err="failed to get container status \"d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4\": rpc error: code = NotFound desc = could not find container \"d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4\": container with ID starting with d0414aa2b3e6832ad5202baf78f4e426aed8a111109b7bd778948a012de0eef4 not found: ID does not exist"
Oct 30 22:51:48.109554 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.109520    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"]
Oct 30 22:51:48.160426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.160386    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt"]
Oct 30 22:51:48.228956 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.228873    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"]
Oct 30 22:51:48.234278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.234254    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw"]
Oct 30 22:51:48.295597 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.295558    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:48.526331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.526244    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:48.645822 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.645782    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05df6d89-3723-4449-b11d-8a2081237bcc" path="/var/lib/kubelet/pods/05df6d89-3723-4449-b11d-8a2081237bcc/volumes"
Oct 30 22:51:48.646182 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:48.646167    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61f736f-bc03-457f-b47b-d65dfc7fcadf" path="/var/lib/kubelet/pods/c61f736f-bc03-457f-b47b-d65dfc7fcadf/volumes"
Oct 30 22:51:49.234483 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:49.234454    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:50.542090 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:51:50.542053    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:51:50.807273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:50.807163    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:51.090545 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:51.090460    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:51.423154 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:51.423118    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"]
Oct 30 22:51:51.473081 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:51.473050    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-40b9a8-nbw62"]
Oct 30 22:51:51.958334 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:51.958297    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:52.645316 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:52.645282    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" path="/var/lib/kubelet/pods/ce3b94c0-ecbe-46cf-b1fd-2b37931edc16/volumes"
Oct 30 22:51:53.062516 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:53.062431    5031 scope.go:122] "RemoveContainer" containerID="75c0860fee63c7bf31b169aa0f8d50af2b0574b7119a8a01897211397ae6f1e3"
Oct 30 22:51:53.071691 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:53.071672    5031 scope.go:122] "RemoveContainer" containerID="11d364244966c54c1c676f055347c4231ccfba7003e0ba5227a2b784abcc3852"
Oct 30 22:51:56.125110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:56.125073    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:51:58.391460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:58.391380    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:58.622043 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:58.622000    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:58.680007 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:58.679925    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"]
Oct 30 22:51:58.680170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:58.680140    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy" containerID="cri-o://68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd" gracePeriod=30
Oct 30 22:51:58.847486 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:51:58.847445    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:00.609574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:00.609538    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:01.988425 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:01.988383    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:52:01.988917 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:01.988463    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:52:03.942347 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.942277    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:52:03.947397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947377    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1fc6ef3-4865-462d-ac04-406d4a717302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc6ef3-4865-462d-ac04-406d4a717302-istiod-ca-cert\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947412    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-certs\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947431    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-socket\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947494 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947473    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-data\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947495    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-kube-api-access-558v9\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-kube-api-access-558v9\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947528    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-envoy\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947635    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-token\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947664    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-credential-socket\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947693    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-podinfo\") pod \"b1fc6ef3-4865-462d-ac04-406d4a717302\" (UID: \"b1fc6ef3-4865-462d-ac04-406d4a717302\") "
Oct 30 22:52:03.947908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947771    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fc6ef3-4865-462d-ac04-406d4a717302-istiod-ca-cert" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:03.947908 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947767    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-certs" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:03.948098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947908    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-data" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:03.948098 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.947927    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-socket" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:03.948240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.948118    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc6ef3-4865-462d-ac04-406d4a717302-istiod-ca-cert\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.948240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.948135    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-certs\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.948240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.948155    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-workload-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.948240 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.948174    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-data\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.948458 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.948349    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-credential-socket" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:03.950049 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.950020    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-kube-api-access-558v9" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "kube-api-access-558v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:03.950164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.950052    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-envoy" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:03.950164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.950058    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-token" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:03.950312 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:03.950269    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-podinfo" pod "b1fc6ef3-4865-462d-ac04-406d4a717302" (UID: "b1fc6ef3-4865-462d-ac04-406d4a717302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:52:04.043680 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.043643    5031 generic.go:356] "Generic (PLEG): container finished" podID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerID="68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd" exitCode=0
Oct 30 22:52:04.043855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.043703    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"
Oct 30 22:52:04.043855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.043732    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" event={"ID":"b1fc6ef3-4865-462d-ac04-406d4a717302","Type":"ContainerDied","Data":"68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd"}
Oct 30 22:52:04.043855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.043800    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l" event={"ID":"b1fc6ef3-4865-462d-ac04-406d4a717302","Type":"ContainerDied","Data":"c4c8db9519403d8e35353b9639536a0352fa6d36d0500a5ca66fb09af36bc2e1"}
Oct 30 22:52:04.043855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.043831    5031 scope.go:122] "RemoveContainer" containerID="68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd"
Oct 30 22:52:04.048671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.048648    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-558v9\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-kube-api-access-558v9\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:04.048671 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.048669    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-envoy\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:04.048875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.048689    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-token\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:04.048875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.048700    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc6ef3-4865-462d-ac04-406d4a717302-credential-socket\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:04.048875 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.048708    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc6ef3-4865-462d-ac04-406d4a717302-istio-podinfo\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:04.057988 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.057962    5031 scope.go:122] "RemoveContainer" containerID="f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7"
Oct 30 22:52:04.071936 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.071909    5031 scope.go:122] "RemoveContainer" containerID="68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd"
Oct 30 22:52:04.072332 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:52:04.072306    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd\": container with ID starting with 68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd not found: ID does not exist" containerID="68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd"
Oct 30 22:52:04.072476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.072342    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd"} err="failed to get container status \"68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd\": rpc error: code = NotFound desc = could not find container \"68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd\": container with ID starting with 68e8063262beca128f6002af982402c0630ff5952779d39d724d2f2a72365acd not found: ID does not exist"
Oct 30 22:52:04.072476 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.072365    5031 scope.go:122] "RemoveContainer" containerID="f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7"
Oct 30 22:52:04.072665 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:52:04.072646    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7\": container with ID starting with f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7 not found: ID does not exist" containerID="f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7"
Oct 30 22:52:04.072745 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.072670    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7"} err="failed to get container status \"f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7\": rpc error: code = NotFound desc = could not find container \"f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7\": container with ID starting with f373a960eb16b89c12c1befded7f0467b5e3de788171f14e6a696e121a0b57b7 not found: ID does not exist"
Oct 30 22:52:04.185330 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.185292    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"]
Oct 30 22:52:04.194968 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.194888    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l"]
Oct 30 22:52:04.644675 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:04.644639    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" path="/var/lib/kubelet/pods/b1fc6ef3-4865-462d-ac04-406d4a717302/volumes"
Oct 30 22:52:06.968651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:06.968608    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:52:17.955949 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:17.955914    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:52:20.621754 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:52:20.621716    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:52:26.095412 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095371    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"]
Oct 30 22:52:26.095858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095844    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy"
Oct 30 22:52:26.095906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095859    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy"
Oct 30 22:52:26.095906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095870    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy"
Oct 30 22:52:26.095906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095875    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy"
Oct 30 22:52:26.095906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095899    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c61f736f-bc03-457f-b47b-d65dfc7fcadf" containerName="payload-pre-processing"
Oct 30 22:52:26.095906 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095905    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61f736f-bc03-457f-b47b-d65dfc7fcadf" containerName="payload-pre-processing"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095926    5031 cpu_manager.go:414] "containerMap: removing container" podUID="05df6d89-3723-4449-b11d-8a2081237bcc" containerName="payload-processing"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095930    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05df6d89-3723-4449-b11d-8a2081237bcc" containerName="payload-processing"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095938    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" containerName="revoke-keys"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.095943    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" containerName="revoke-keys"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.096004    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61f736f-bc03-457f-b47b-d65dfc7fcadf" containerName="payload-pre-processing"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.096018    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="05df6d89-3723-4449-b11d-8a2081237bcc" containerName="payload-processing"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.096025    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3b94c0-ecbe-46cf-b1fd-2b37931edc16" containerName="revoke-keys"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.096030    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy"
Oct 30 22:52:26.096075 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.096036    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1fc6ef3-4865-462d-ac04-406d4a717302" containerName="istio-proxy"
Oct 30 22:52:26.099378 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.099360    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.162884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.162852    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"]
Oct 30 22:52:26.207781 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.207734    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.207999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.207865    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.208072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.208000    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphpp\" (UniqueName: \"kubernetes.io/projected/9633e791-aeab-471a-8cc0-dc3b789fc3c5-kube-api-access-xphpp\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.209760 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.209737    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:26.209947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.209772    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:26.209947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.209837    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:26.209947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.209923    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfs8q\""
Oct 30 22:52:26.309041 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.309005    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.309278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.309084    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphpp\" (UniqueName: \"kubernetes.io/projected/9633e791-aeab-471a-8cc0-dc3b789fc3c5-kube-api-access-xphpp\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.309278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.309122    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.309789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.309755    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.311636 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.311616    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.374009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.373933    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphpp\" (UniqueName: \"kubernetes.io/projected/9633e791-aeab-471a-8cc0-dc3b789fc3c5-kube-api-access-xphpp\") pod \"maas-api-revoke-sub-f03629db-xgd28\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.414120 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.414080    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:26.799202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:26.799167    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"]
Oct 30 22:52:26.800701 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:52:26.800674    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9633e791_aeab_471a_8cc0_dc3b789fc3c5.slice/crio-61dc543237db05a17c959a09e05345c7aca485bc35f6a2d5963da1b7869cc990 WatchSource:0}: Error finding container 61dc543237db05a17c959a09e05345c7aca485bc35f6a2d5963da1b7869cc990: Status 404 returned error can't find the container with id 61dc543237db05a17c959a09e05345c7aca485bc35f6a2d5963da1b7869cc990
Oct 30 22:52:27.221805 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:27.221766    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28" event={"ID":"9633e791-aeab-471a-8cc0-dc3b789fc3c5","Type":"ContainerStarted","Data":"b487a92b7fa77da19ef2d6ef0c0b78e8168d2ca3d20bc074fef1a3db78ce750b"}
Oct 30 22:52:27.222309 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:27.221831    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28" event={"ID":"9633e791-aeab-471a-8cc0-dc3b789fc3c5","Type":"ContainerStarted","Data":"61dc543237db05a17c959a09e05345c7aca485bc35f6a2d5963da1b7869cc990"}
Oct 30 22:52:27.353855 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:27.353790    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28" podStartSLOduration=1.353777171 podStartE2EDuration="1.353777171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:27.351468383 +0000 UTC m=+3758.875611147" watchObservedRunningTime="2026-10-30 22:52:27.353777171 +0000 UTC m=+3758.877920007"
Oct 30 22:52:28.229401 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:28.229360    5031 generic.go:356] "Generic (PLEG): container finished" podID="9633e791-aeab-471a-8cc0-dc3b789fc3c5" containerID="b487a92b7fa77da19ef2d6ef0c0b78e8168d2ca3d20bc074fef1a3db78ce750b" exitCode=0
Oct 30 22:52:28.229851 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:28.229458    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28" event={"ID":"9633e791-aeab-471a-8cc0-dc3b789fc3c5","Type":"ContainerDied","Data":"b487a92b7fa77da19ef2d6ef0c0b78e8168d2ca3d20bc074fef1a3db78ce750b"}
Oct 30 22:52:29.393298 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.393267    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:29.542684 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.542579    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-openshift-ca\") pod \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") "
Oct 30 22:52:29.542882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.542719    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-xks-ca\") pod \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") "
Oct 30 22:52:29.542882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.542752    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9633e791-aeab-471a-8cc0-dc3b789fc3c5-kube-api-access-xphpp\" (UniqueName: \"kubernetes.io/projected/9633e791-aeab-471a-8cc0-dc3b789fc3c5-kube-api-access-xphpp\") pod \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\" (UID: \"9633e791-aeab-471a-8cc0-dc3b789fc3c5\") "
Oct 30 22:52:29.543110 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.543080    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-openshift-ca" pod "9633e791-aeab-471a-8cc0-dc3b789fc3c5" (UID: "9633e791-aeab-471a-8cc0-dc3b789fc3c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:29.545258 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.545199    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-xks-ca" pod "9633e791-aeab-471a-8cc0-dc3b789fc3c5" (UID: "9633e791-aeab-471a-8cc0-dc3b789fc3c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:29.545357 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.545247    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9633e791-aeab-471a-8cc0-dc3b789fc3c5-kube-api-access-xphpp" pod "9633e791-aeab-471a-8cc0-dc3b789fc3c5" (UID: "9633e791-aeab-471a-8cc0-dc3b789fc3c5"). InnerVolumeSpecName "kube-api-access-xphpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:29.644515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.644479    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.644515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.644513    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9633e791-aeab-471a-8cc0-dc3b789fc3c5-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.644515 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:29.644523    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xphpp\" (UniqueName: \"kubernetes.io/projected/9633e791-aeab-471a-8cc0-dc3b789fc3c5-kube-api-access-xphpp\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:30.243176 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:30.243144    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28"
Oct 30 22:52:30.243381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:30.243177    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xgd28" event={"ID":"9633e791-aeab-471a-8cc0-dc3b789fc3c5","Type":"ContainerDied","Data":"61dc543237db05a17c959a09e05345c7aca485bc35f6a2d5963da1b7869cc990"}
Oct 30 22:52:30.243381 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:30.243246    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61dc543237db05a17c959a09e05345c7aca485bc35f6a2d5963da1b7869cc990"
Oct 30 22:52:39.056916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:39.056869    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:52:40.705368 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:40.705333    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:45.319821 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.319764    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"]
Oct 30 22:52:45.321234 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.321189    5031 cpu_manager.go:414] "containerMap: removing container" podUID="9633e791-aeab-471a-8cc0-dc3b789fc3c5" containerName="revoke-keys"
Oct 30 22:52:45.321398 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.321379    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9633e791-aeab-471a-8cc0-dc3b789fc3c5" containerName="revoke-keys"
Oct 30 22:52:45.321654 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.321636    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="9633e791-aeab-471a-8cc0-dc3b789fc3c5" containerName="revoke-keys"
Oct 30 22:52:45.328549 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.328521    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.449277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.405546    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.449277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.405658    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.449277 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.405732    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvvd\" (UniqueName: \"kubernetes.io/projected/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-kube-api-access-grvvd\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.482237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.451100    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:45.482237 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.451307    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfs8q\""
Oct 30 22:52:45.487340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.486000    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:45.487340 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.486082    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:45.504608 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.504458    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"]
Oct 30 22:52:45.510244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.506759    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.510244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.506848    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.510244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.506913    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvvd\" (UniqueName: \"kubernetes.io/projected/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-kube-api-access-grvvd\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.510244 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.508168    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.596264 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.595935    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.596657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.596627    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvvd\" (UniqueName: \"kubernetes.io/projected/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-kube-api-access-grvvd\") pod \"maas-api-revoke-sub-0165d7d2-xckfb\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:45.888649 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:45.888551    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:46.437521 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:46.433755    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"]
Oct 30 22:52:46.454731 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:52:46.450512    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb91df8_5698_4f92_8c4f_ba5aa8d202e1.slice/crio-f13d5d7627fda0a272c4bb1f354038fb0c173d832573d7d9c6fea4728f558d63 WatchSource:0}: Error finding container f13d5d7627fda0a272c4bb1f354038fb0c173d832573d7d9c6fea4728f558d63: Status 404 returned error can't find the container with id f13d5d7627fda0a272c4bb1f354038fb0c173d832573d7d9c6fea4728f558d63
Oct 30 22:52:46.578426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:46.570283    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb" event={"ID":"edb91df8-5698-4f92-8c4f-ba5aa8d202e1","Type":"ContainerStarted","Data":"f13d5d7627fda0a272c4bb1f354038fb0c173d832573d7d9c6fea4728f558d63"}
Oct 30 22:52:47.577947 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:47.577914    5031 generic.go:356] "Generic (PLEG): container finished" podID="edb91df8-5698-4f92-8c4f-ba5aa8d202e1" containerID="80f2600011cbb2d9869e215ea6e4dd9d686671923638f6b6026a2a42a660a78c" exitCode=0
Oct 30 22:52:47.578364 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:47.577976    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb" event={"ID":"edb91df8-5698-4f92-8c4f-ba5aa8d202e1","Type":"ContainerDied","Data":"80f2600011cbb2d9869e215ea6e4dd9d686671923638f6b6026a2a42a660a78c"}
Oct 30 22:52:48.742948 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.742922    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:48.793960 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.793925    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-xks-ca\") pod \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") "
Oct 30 22:52:48.794170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.793974    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-openshift-ca\") pod \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") "
Oct 30 22:52:48.794170 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.794091    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-kube-api-access-grvvd\" (UniqueName: \"kubernetes.io/projected/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-kube-api-access-grvvd\") pod \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\" (UID: \"edb91df8-5698-4f92-8c4f-ba5aa8d202e1\") "
Oct 30 22:52:48.794420 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.794397    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-openshift-ca" pod "edb91df8-5698-4f92-8c4f-ba5aa8d202e1" (UID: "edb91df8-5698-4f92-8c4f-ba5aa8d202e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:48.796224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.796190    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-xks-ca" pod "edb91df8-5698-4f92-8c4f-ba5aa8d202e1" (UID: "edb91df8-5698-4f92-8c4f-ba5aa8d202e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:48.796365 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.796342    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-kube-api-access-grvvd" pod "edb91df8-5698-4f92-8c4f-ba5aa8d202e1" (UID: "edb91df8-5698-4f92-8c4f-ba5aa8d202e1"). InnerVolumeSpecName "kube-api-access-grvvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:48.895884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.895845    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvvd\" (UniqueName: \"kubernetes.io/projected/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-kube-api-access-grvvd\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:48.895884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.895877    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:48.895884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:48.895887    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb91df8-5698-4f92-8c4f-ba5aa8d202e1-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.591550 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:49.591515    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb"
Oct 30 22:52:49.591747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:49.591555    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xckfb" event={"ID":"edb91df8-5698-4f92-8c4f-ba5aa8d202e1","Type":"ContainerDied","Data":"f13d5d7627fda0a272c4bb1f354038fb0c173d832573d7d9c6fea4728f558d63"}
Oct 30 22:52:49.591747 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:49.591612    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13d5d7627fda0a272c4bb1f354038fb0c173d832573d7d9c6fea4728f558d63"
Oct 30 22:52:50.540989 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:52:50.540946    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:52:57.112718 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.112683    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"]
Oct 30 22:52:57.113300 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.113282    5031 cpu_manager.go:414] "containerMap: removing container" podUID="edb91df8-5698-4f92-8c4f-ba5aa8d202e1" containerName="revoke-keys"
Oct 30 22:52:57.113389 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.113301    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb91df8-5698-4f92-8c4f-ba5aa8d202e1" containerName="revoke-keys"
Oct 30 22:52:57.113457 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.113430    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb91df8-5698-4f92-8c4f-ba5aa8d202e1" containerName="revoke-keys"
Oct 30 22:52:57.116587 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.116567    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.176673 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.176631    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"]
Oct 30 22:52:57.186082 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.186040    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnm8k\" (UniqueName: \"kubernetes.io/projected/fcb45083-eb46-46a8-a946-ef400bd65f7d-kube-api-access-gnm8k\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.186284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.186130    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.186284 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.186188    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.223260 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.223194    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:57.223482 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.223458    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:57.223570 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.223549    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:57.223648 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.223559    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfs8q\""
Oct 30 22:52:57.287439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.287393    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.287630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.287550    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnm8k\" (UniqueName: \"kubernetes.io/projected/fcb45083-eb46-46a8-a946-ef400bd65f7d-kube-api-access-gnm8k\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.287630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.287603    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.288273 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.288248    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.290397 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.290375    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.351564 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.351524    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnm8k\" (UniqueName: \"kubernetes.io/projected/fcb45083-eb46-46a8-a946-ef400bd65f7d-kube-api-access-gnm8k\") pod \"maas-api-revoke-sub-6b19338f-fnlcb\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.430758 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.430660    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:52:57.813676 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:57.813649    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"]
Oct 30 22:52:57.815176 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:52:57.815142    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb45083_eb46_46a8_a946_ef400bd65f7d.slice/crio-6f6ba7c89cfee5cbabefabf26ecfb81da625c23039d1d311aac66d7ab316d753 WatchSource:0}: Error finding container 6f6ba7c89cfee5cbabefabf26ecfb81da625c23039d1d311aac66d7ab316d753: Status 404 returned error can't find the container with id 6f6ba7c89cfee5cbabefabf26ecfb81da625c23039d1d311aac66d7ab316d753
Oct 30 22:52:58.657306 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:58.657269    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb" event={"ID":"fcb45083-eb46-46a8-a946-ef400bd65f7d","Type":"ContainerStarted","Data":"72e415b35a0694d7aea52938423e2b8c475577761040630c0cb5f5616d75e638"}
Oct 30 22:52:58.657750 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:58.657334    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb" event={"ID":"fcb45083-eb46-46a8-a946-ef400bd65f7d","Type":"ContainerStarted","Data":"6f6ba7c89cfee5cbabefabf26ecfb81da625c23039d1d311aac66d7ab316d753"}
Oct 30 22:52:58.778942 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:58.778884    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb" podStartSLOduration=1.778871052 podStartE2EDuration="1.778871052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:58.777191192 +0000 UTC m=+3790.301333956" watchObservedRunningTime="2026-10-30 22:52:58.778871052 +0000 UTC m=+3790.303013816"
Oct 30 22:52:59.665464 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:59.665428    5031 generic.go:356] "Generic (PLEG): container finished" podID="fcb45083-eb46-46a8-a946-ef400bd65f7d" containerID="72e415b35a0694d7aea52938423e2b8c475577761040630c0cb5f5616d75e638" exitCode=0
Oct 30 22:52:59.665904 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:52:59.665501    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb" event={"ID":"fcb45083-eb46-46a8-a946-ef400bd65f7d","Type":"ContainerDied","Data":"72e415b35a0694d7aea52938423e2b8c475577761040630c0cb5f5616d75e638"}
Oct 30 22:53:00.828388 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:00.828355    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:53:00.921858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:00.921827    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-openshift-ca\") pod \"fcb45083-eb46-46a8-a946-ef400bd65f7d\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") "
Oct 30 22:53:00.922100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:00.921931    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-xks-ca\") pod \"fcb45083-eb46-46a8-a946-ef400bd65f7d\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") "
Oct 30 22:53:00.922100 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:00.921979    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb45083-eb46-46a8-a946-ef400bd65f7d-kube-api-access-gnm8k\" (UniqueName: \"kubernetes.io/projected/fcb45083-eb46-46a8-a946-ef400bd65f7d-kube-api-access-gnm8k\") pod \"fcb45083-eb46-46a8-a946-ef400bd65f7d\" (UID: \"fcb45083-eb46-46a8-a946-ef400bd65f7d\") "
Oct 30 22:53:00.922331 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:00.922300    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-openshift-ca" pod "fcb45083-eb46-46a8-a946-ef400bd65f7d" (UID: "fcb45083-eb46-46a8-a946-ef400bd65f7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:00.924400 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:00.924339    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-xks-ca" pod "fcb45083-eb46-46a8-a946-ef400bd65f7d" (UID: "fcb45083-eb46-46a8-a946-ef400bd65f7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:00.924519 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:00.924458    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb45083-eb46-46a8-a946-ef400bd65f7d-kube-api-access-gnm8k" pod "fcb45083-eb46-46a8-a946-ef400bd65f7d" (UID: "fcb45083-eb46-46a8-a946-ef400bd65f7d"). InnerVolumeSpecName "kube-api-access-gnm8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:01.022810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:01.022774    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-openshift-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:01.022810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:01.022808    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb45083-eb46-46a8-a946-ef400bd65f7d-maas-api-xks-ca\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:01.022810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:01.022819    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnm8k\" (UniqueName: \"kubernetes.io/projected/fcb45083-eb46-46a8-a946-ef400bd65f7d-kube-api-access-gnm8k\") on node \"ip-10-0-141-147.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:01.027997 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:01.027971    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:01.678864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:01.678835    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb"
Oct 30 22:53:01.678864 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:01.678857    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fnlcb" event={"ID":"fcb45083-eb46-46a8-a946-ef400bd65f7d","Type":"ContainerDied","Data":"6f6ba7c89cfee5cbabefabf26ecfb81da625c23039d1d311aac66d7ab316d753"}
Oct 30 22:53:01.679096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:01.678906    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f6ba7c89cfee5cbabefabf26ecfb81da625c23039d1d311aac66d7ab316d753"
Oct 30 22:53:09.247810 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:09.247771    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bbc6dbb5d-4jll5_faf7f082-2144-4538-959e-5f30b01c7aa1/authorino/0.log"
Oct 30 22:53:09.974096 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:09.974056    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j8xxp"]
Oct 30 22:53:12.204293 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:12.204243    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kkskg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:53:12.204738 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:12.204328    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg" podUID="de2775d7-88bd-440d-8034-5a7412a22128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:53:16.057878 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:16.057828    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"]
Oct 30 22:53:16.108297 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:16.108261    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56c627-6fdqs"]
Oct 30 22:53:16.645117 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:16.645084    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a69a495-3a91-46ed-abe8-38b96966cd8b" path="/var/lib/kubelet/pods/1a69a495-3a91-46ed-abe8-38b96966cd8b/volumes"
Oct 30 22:53:19.866402 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:19.866370    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-65d9c54ff9-bncj2_128bd426-3a53-4314-ad3f-a76d97b84950/manager/0.log"
Oct 30 22:53:20.541583 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:53:20.541548    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:53:22.180314 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:22.180281    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-92hjx_f5932308-a52b-42d0-9706-2776b7e7fea4/server/0.log"
Oct 30 22:53:22.411440 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:22.411404    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9fbbk_6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1/manager/0.log"
Oct 30 22:53:24.189092 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:24.189060    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-xckfb_edb91df8-5698-4f92-8c4f-ba5aa8d202e1/revoke-keys/0.log"
Oct 30 22:53:24.412657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:24.412621    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fnlcb_fcb45083-eb46-46a8-a946-ef400bd65f7d/revoke-keys/0.log"
Oct 30 22:53:24.629224 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:24.629189    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xgd28_9633e791-aeab-471a-8cc0-dc3b789fc3c5/revoke-keys/0.log"
Oct 30 22:53:26.374583 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:26.374547    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bbc6dbb5d-4jll5_faf7f082-2144-4538-959e-5f30b01c7aa1/authorino/0.log"
Oct 30 22:53:26.847802 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:26.847769    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dmnft_5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e/manager/0.log"
Oct 30 22:53:27.088497 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:27.088468    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ms8lr_b583470d-0cc1-4196-8225-4ffcb972f282/manager/0.log"
Oct 30 22:53:27.546411 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:27.546332    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zdsf7_191e15f8-408c-4914-a9ee-f46f43bb2b6c/manager/0.log"
Oct 30 22:53:27.901920 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:27.901893    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j8xxp_98377d7d-c750-4134-8611-00d9103d85b0/limitador/0.log"
Oct 30 22:53:28.135164 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:28.135136    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cw5tc_51231b17-e61d-414b-a7d9-3ef076e2fcf5/manager/0.log"
Oct 30 22:53:28.948935 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:28.948909    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8_eaeed454-d15a-4895-8a38-bef8ac996847/istio-proxy/0.log"
Oct 30 22:53:30.783657 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:30.783628    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-kkskg_de2775d7-88bd-440d-8034-5a7412a22128/istio-proxy/1.log"
Oct 30 22:53:31.110651 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:31.110625    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nqjll_b622abc4-5b50-4f1b-84cc-30edabc33258/istio-proxy/0.log"
Oct 30 22:53:31.327584 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:31.327553    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-t54jj_cc3fa4fe-d521-4a83-b82d-27bf5dda0f64/istio-proxy/0.log"
Oct 30 22:53:32.795370 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:32.795335    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d_869e56ec-0645-4744-9763-6c8548f77ad8/istio-proxy/0.log"
Oct 30 22:53:33.068099 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:33.068023    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f82bg_2007da67-21d3-4b4a-aa73-69ad80f82250/payload-pre-processing/0.log"
Oct 30 22:53:33.292475 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:33.292429    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wztzx_3f526c67-8dfe-4b2e-9c8f-42aa57efebec/payload-processing/0.log"
Oct 30 22:53:33.517727 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:33.517697    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d48b6d977-dstqd_47bc5387-3f28-4d19-9f51-cdd6e298896f/router/0.log"
Oct 30 22:53:34.031140 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:34.031104    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-kkskg_de2775d7-88bd-440d-8034-5a7412a22128/istio-proxy/0.log"
Oct 30 22:53:34.601344 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:34.601319    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn_8a6d5c6a-ec67-4875-960c-916306762084/main/0.log"
Oct 30 22:53:35.087927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:35.087845    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c4r4m_ed248874-db33-429b-b509-2a75e3e72304/main/0.log"
Oct 30 22:53:35.354349 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:35.354321    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-5grw9_1368ca02-43fe-4181-b9e4-933df7a9665e/main/0.log"
Oct 30 22:53:35.575696 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:35.575663    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv_52bfc710-4ddf-4e96-a333-ead674e668ed/main/0.log"
Oct 30 22:53:35.803786 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:35.803710    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-gdzbd_06f5935f-a491-4b89-a3a5-bec3aa0f8b26/main/0.log"
Oct 30 22:53:37.210295 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:37.210265    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bbc6dbb5d-4jll5_faf7f082-2144-4538-959e-5f30b01c7aa1/authorino/0.log"
Oct 30 22:53:46.358531 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:46.358499    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-65d9c54ff9-bncj2_128bd426-3a53-4314-ad3f-a76d97b84950/manager/0.log"
Oct 30 22:53:48.065789 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:48.065759    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-92hjx_f5932308-a52b-42d0-9706-2776b7e7fea4/server/0.log"
Oct 30 22:53:48.301129 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:48.301094    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9fbbk_6c6d4cef-2c1a-4f0f-bfe4-5405883c1cb1/manager/0.log"
Oct 30 22:53:49.938703 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:49.938676    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-xckfb_edb91df8-5698-4f92-8c4f-ba5aa8d202e1/revoke-keys/0.log"
Oct 30 22:53:50.151681 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:50.151652    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fnlcb_fcb45083-eb46-46a8-a946-ef400bd65f7d/revoke-keys/0.log"
Oct 30 22:53:50.376883 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:50.376859    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xgd28_9633e791-aeab-471a-8cc0-dc3b789fc3c5/revoke-keys/0.log"
Oct 30 22:53:50.541733 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:53:50.541696    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:53:51.825556 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:51.825520    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bbc6dbb5d-4jll5_faf7f082-2144-4538-959e-5f30b01c7aa1/authorino/0.log"
Oct 30 22:53:52.295009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:52.294978    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dmnft_5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e/manager/0.log"
Oct 30 22:53:52.541650 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:52.541615    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ms8lr_b583470d-0cc1-4196-8225-4ffcb972f282/manager/0.log"
Oct 30 22:53:53.004630 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:53.004599    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zdsf7_191e15f8-408c-4914-a9ee-f46f43bb2b6c/manager/0.log"
Oct 30 22:53:53.275672 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:53.275588    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j8xxp_98377d7d-c750-4134-8611-00d9103d85b0/limitador/0.log"
Oct 30 22:53:53.502999 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:53.502972    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cw5tc_51231b17-e61d-414b-a7d9-3ef076e2fcf5/manager/0.log"
Oct 30 22:53:54.084231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:54.084175    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8_eaeed454-d15a-4895-8a38-bef8ac996847/istio-proxy/0.log"
Oct 30 22:53:55.920783 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:55.920749    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-kkskg_de2775d7-88bd-440d-8034-5a7412a22128/istio-proxy/1.log"
Oct 30 22:53:56.247202 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:56.247121    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nqjll_b622abc4-5b50-4f1b-84cc-30edabc33258/istio-proxy/0.log"
Oct 30 22:53:56.480804 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:56.480771    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-t54jj_cc3fa4fe-d521-4a83-b82d-27bf5dda0f64/istio-proxy/0.log"
Oct 30 22:53:57.902614 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:57.902584    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d_869e56ec-0645-4744-9763-6c8548f77ad8/istio-proxy/0.log"
Oct 30 22:53:58.172957 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:58.172875    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f82bg_2007da67-21d3-4b4a-aa73-69ad80f82250/payload-pre-processing/0.log"
Oct 30 22:53:58.386335 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:58.386294    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wztzx_3f526c67-8dfe-4b2e-9c8f-42aa57efebec/payload-processing/0.log"
Oct 30 22:53:58.619456 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:58.619432    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d48b6d977-dstqd_47bc5387-3f28-4d19-9f51-cdd6e298896f/router/0.log"
Oct 30 22:53:59.217122 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:59.217092    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l8trn_8a6d5c6a-ec67-4875-960c-916306762084/main/0.log"
Oct 30 22:53:59.744984 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:53:59.744951    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c4r4m_ed248874-db33-429b-b509-2a75e3e72304/main/0.log"
Oct 30 22:54:00.013408 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:00.013331    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-5grw9_1368ca02-43fe-4181-b9e4-933df7a9665e/main/0.log"
Oct 30 22:54:00.237574 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:00.237546    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lf9vv_52bfc710-4ddf-4e96-a333-ead674e668ed/main/0.log"
Oct 30 22:54:00.463128 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:00.463095    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-gdzbd_06f5935f-a491-4b89-a3a5-bec3aa0f8b26/main/0.log"
Oct 30 22:54:01.335915 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:01.335881    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:20.540667 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:54:20.540624    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:54:29.715919 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:29.715879    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wztzx_3f526c67-8dfe-4b2e-9c8f-42aa57efebec/payload-processing/0.log"
Oct 30 22:54:29.943231 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:29.943186    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wztzx_3f526c67-8dfe-4b2e-9c8f-42aa57efebec/payload-processing/0.log"
Oct 30 22:54:34.070410 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:34.070375    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:46.449072 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:46.449041    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:50.188399 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:50.188366    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:54:50.209235 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:50.209183    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fs49h_d07ee097-855f-4598-a947-2268077bac00/console-operator/1.log"
Oct 30 22:54:50.540893 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:54:50.540817    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:54:53.133782 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:54:53.133747    5031 scope.go:122] "RemoveContainer" containerID="524b04a941804d67a8e438572d1f7899949b136612b8dea405f5dd0f4dd49658"
Oct 30 22:55:01.681715 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:01.681680    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:06.984153 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:06.984124    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wztzx_3f526c67-8dfe-4b2e-9c8f-42aa57efebec/payload-processing/0.log"
Oct 30 22:55:07.209965 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.209939    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wztzx_3f526c67-8dfe-4b2e-9c8f-42aa57efebec/payload-processing/0.log"
Oct 30 22:55:07.546418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.546382    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8jvzx/must-gather-g65tt"]
Oct 30 22:55:07.546849 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.546838    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fcb45083-eb46-46a8-a946-ef400bd65f7d" containerName="revoke-keys"
Oct 30 22:55:07.546897 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.546851    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb45083-eb46-46a8-a946-ef400bd65f7d" containerName="revoke-keys"
Oct 30 22:55:07.546934 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.546926    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb45083-eb46-46a8-a946-ef400bd65f7d" containerName="revoke-keys"
Oct 30 22:55:07.551721 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.551700    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:07.619677 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.619654    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:07.656163 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.656138    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8jvzx\"/\"default-dockercfg-sdt8t\""
Oct 30 22:55:07.656419 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.656405    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jvzx\"/\"openshift-service-ca.crt\""
Oct 30 22:55:07.656518 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.656467    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jvzx\"/\"kube-root-ca.crt\""
Oct 30 22:55:07.659302 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.659282    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/must-gather-g65tt"]
Oct 30 22:55:07.690243 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.690183    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gr5g\" (UniqueName: \"kubernetes.io/projected/cf28cb09-5147-46b5-818f-675a719cee91-kube-api-access-8gr5g\") pod \"must-gather-g65tt\" (UID: \"cf28cb09-5147-46b5-818f-675a719cee91\") " pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:07.690426 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.690399    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf28cb09-5147-46b5-818f-675a719cee91-must-gather-output\") pod \"must-gather-g65tt\" (UID: \"cf28cb09-5147-46b5-818f-675a719cee91\") " pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:07.791439 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.791408    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gr5g\" (UniqueName: \"kubernetes.io/projected/cf28cb09-5147-46b5-818f-675a719cee91-kube-api-access-8gr5g\") pod \"must-gather-g65tt\" (UID: \"cf28cb09-5147-46b5-818f-675a719cee91\") " pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:07.791591 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.791471    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf28cb09-5147-46b5-818f-675a719cee91-must-gather-output\") pod \"must-gather-g65tt\" (UID: \"cf28cb09-5147-46b5-818f-675a719cee91\") " pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:07.791784 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.791758    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf28cb09-5147-46b5-818f-675a719cee91-must-gather-output\") pod \"must-gather-g65tt\" (UID: \"cf28cb09-5147-46b5-818f-675a719cee91\") " pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:07.851694 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.851642    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:07.854890 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.854866    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gr5g\" (UniqueName: \"kubernetes.io/projected/cf28cb09-5147-46b5-818f-675a719cee91-kube-api-access-8gr5g\") pod \"must-gather-g65tt\" (UID: \"cf28cb09-5147-46b5-818f-675a719cee91\") " pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:07.868830 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:07.868808    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/must-gather-g65tt"
Oct 30 22:55:08.258003 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:08.257980    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/must-gather-g65tt"]
Oct 30 22:55:08.264607 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:08.264581    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:55:08.482109 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:08.482071    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/must-gather-g65tt" event={"ID":"cf28cb09-5147-46b5-818f-675a719cee91","Type":"ContainerStarted","Data":"02cc5ecf6c7179eb8af2814cb63c314e3cb4375fff939c2a21803ec665e9c8f8"}
Oct 30 22:55:10.496418 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:10.496376    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/must-gather-g65tt" event={"ID":"cf28cb09-5147-46b5-818f-675a719cee91","Type":"ContainerStarted","Data":"fb5b4358290c32d87536633c9e007065202d62125b865b4ee4e2dc43375db6a6"}
Oct 30 22:55:10.500498 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:10.496430    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/must-gather-g65tt" event={"ID":"cf28cb09-5147-46b5-818f-675a719cee91","Type":"ContainerStarted","Data":"e855ea2c1fdc1d48b77e3eac84cfe92f453660bc86c3e9a515f510061e52d5b6"}
Oct 30 22:55:10.618858 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:10.618802    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8jvzx/must-gather-g65tt" podStartSLOduration=2.304563955 podStartE2EDuration="3.618786121s" totalImagesPullingTime="1.314222166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:10.615492519 +0000 UTC m=+3922.139635275" watchObservedRunningTime="2026-10-30 22:55:10.618786121 +0000 UTC m=+3922.142928885"
Oct 30 22:55:17.762616 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:17.762575    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jl8q2_2f3d72c3-e44c-46be-bbd5-5ecf63f2aad5/global-pull-secret-syncer/0.log"
Oct 30 22:55:19.070927 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:19.070882    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jqx4n_1fdc58d1-11b9-4879-8879-8909f186bbe5/connection-checker/0.log"
Oct 30 22:55:19.520239 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:19.519366    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-m87n9_920770c6-fd18-41d7-95f6-a55c195361e6/connection-checker/0.log"
Oct 30 22:55:19.953916 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:19.953886    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w9w7t_9aaad021-a27a-4ee3-97d7-71fbaa1aff87/connection-checker/0.log"
Oct 30 22:55:20.427256 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:20.427223    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log"
Oct 30 22:55:20.552242 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:55:20.551784    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:55:22.848371 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:22.848329    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-147.us-west-1.compute.internal_46aa86434e4345d68e5cda43f3103b52/haproxy/0.log"
Oct 30 22:55:26.777922 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:26.777888    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bbc6dbb5d-4jll5_faf7f082-2144-4538-959e-5f30b01c7aa1/authorino/0.log"
Oct 30 22:55:27.109717 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.109679    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dmnft_5fb8d3ea-7ffc-4afe-9e2a-f549e3731b8e/manager/0.log"
Oct 30 22:55:27.260695 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.260661    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"]
Oct 30 22:55:27.267859 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.267835    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.332009 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.331973    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpw8\" (UniqueName: \"kubernetes.io/projected/72bec965-2087-4f2d-b70f-ba64039a9398-kube-api-access-fwpw8\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.332186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.332017    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-lib-modules\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.332186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.332125    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-podres\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.332186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.332140    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-proc\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.332186 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.332162    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-sys\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.348623 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.348585    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"]
Oct 30 22:55:27.433798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433703    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-sys\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.433798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433773    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpw8\" (UniqueName: \"kubernetes.io/projected/72bec965-2087-4f2d-b70f-ba64039a9398-kube-api-access-fwpw8\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.434046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433803    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-lib-modules\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.434046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433862    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-sys\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.434046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433875    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-podres\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.434046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433890    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-proc\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.434046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433949    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-lib-modules\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.434046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433970    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-proc\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.434046 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.433987    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72bec965-2087-4f2d-b70f-ba64039a9398-podres\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.497615 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.497579    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpw8\" (UniqueName: \"kubernetes.io/projected/72bec965-2087-4f2d-b70f-ba64039a9398-kube-api-access-fwpw8\") pod \"perf-node-gather-daemonset-smqld\" (UID: \"72bec965-2087-4f2d-b70f-ba64039a9398\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.546029 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.545998    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ms8lr_b583470d-0cc1-4196-8225-4ffcb972f282/manager/0.log"
Oct 30 22:55:27.583882 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.583847    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:27.979152 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:27.979126    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"]
Oct 30 22:55:27.980240 ip-10-0-141-147 kubenswrapper[5031]: W1030 22:55:27.980186    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod72bec965_2087_4f2d_b70f_ba64039a9398.slice/crio-d2e2ff90dd752fc409095afdb42c53aed61e1e09292101c8bf048af5d4049c9a WatchSource:0}: Error finding container d2e2ff90dd752fc409095afdb42c53aed61e1e09292101c8bf048af5d4049c9a: Status 404 returned error can't find the container with id d2e2ff90dd752fc409095afdb42c53aed61e1e09292101c8bf048af5d4049c9a
Oct 30 22:55:28.434360 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:28.434323    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zdsf7_191e15f8-408c-4914-a9ee-f46f43bb2b6c/manager/0.log"
Oct 30 22:55:28.680798 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:28.680759    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld" event={"ID":"72bec965-2087-4f2d-b70f-ba64039a9398","Type":"ContainerStarted","Data":"dae4eebe013d1c16da3fc17d38708489dcb3c678d425271b89a2558b0e58970a"}
Oct 30 22:55:28.680982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:28.680822    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld" event={"ID":"72bec965-2087-4f2d-b70f-ba64039a9398","Type":"ContainerStarted","Data":"d2e2ff90dd752fc409095afdb42c53aed61e1e09292101c8bf048af5d4049c9a"}
Oct 30 22:55:28.680982 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:28.680854    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:28.801438 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:28.801332    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld" podStartSLOduration=1.80131856 podStartE2EDuration="1.80131856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:28.79994323 +0000 UTC m=+3940.324085995" watchObservedRunningTime="2026-10-30 22:55:28.80131856 +0000 UTC m=+3940.325461323"
Oct 30 22:55:28.848283 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:28.848257    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j8xxp_98377d7d-c750-4134-8611-00d9103d85b0/limitador/0.log"
Oct 30 22:55:29.317629 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:29.317576    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cw5tc_51231b17-e61d-414b-a7d9-3ef076e2fcf5/manager/0.log"
Oct 30 22:55:34.703884 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:34.703852    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-smqld"
Oct 30 22:55:35.021079 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:35.020991    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6spwz_bf193224-26cc-4cd6-8aa5-2424e1212c2a/cluster-monitoring-operator/0.log"
Oct 30 22:55:38.171911 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:38.171873    5031 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d77b4_7ae2e290-c982-4b8a-a815-d3babf2679ba/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:40.423189 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:40.423137    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssbjl_79a31e89-7e08-4133-aa3c-4b254b273452/node-exporter/0.log"
Oct 30 22:55:40.848278 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:40.848248    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssbjl_79a31e89-7e08-4133-aa3c-4b254b273452/kube-rbac-proxy/0.log"
Oct 30 22:55:41.275460 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:41.275392    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssbjl_79a31e89-7e08-4133-aa3c-4b254b273452/init-textfile/0.log"
Oct 30 22:55:50.542280 ip-10-0-141-147 kubenswrapper[5031]: E1030 22:55:50.542202    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvqh6" containerName="extractor"
Oct 30 22:55:50.844452 ip-10-0-141-147 kubenswrapper[5031]: I1030 22:55:50.844424    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8_eaeed454-d15a-4895-8a38-bef8ac996847/istio-proxy/0.log"